Technical  Report  1234 


Effects  of  Input  Device  and  Latency  on  Performance 
While  Training  to  Pilot  a  Simulated  Micro-Unmanned 

Aerial  Vehicle 


Deborah  R.  Billings 

University  of  Central  Florida 
Consortium  Research  Fellows  Program 

Paula  J.  Durlach 

U.  S.  Army  Research  Institute 


July  2008 


20090209214 


United  States  Army  Research  Institute 
for  the  Behavioral  and  Social  Sciences 


Approved  for  public  release;  distribution  is  unlimited. 


U.S.  Army  Research  Institute 

for  the  Behavioral  and  Social  Sciences 


A  Directorate  of  the  Department  of  the  Army 
Deputy  Chief  of  Staff,  G1 


BARBARA  A.  BLACK,  Ph.D. 
Research  Program  Manager 
Training  and  Leader  Development 


MICHELLE  SAMS,  PhD. 
Director 


Technical  review  by 

Jessie  Chen,  U.S.  Army  Research  Laboratory 
John  Barnett,  U.S.  Army  Research  Institute 


NOTICES 

DISTRIBUTION:  Primary  distribution  of  this  Research  Report  has  been  made  by  ARI. 
Please  address  correspondence  concerning  distribution  of  reports  to:  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences,  Attn:  DAPC-ARI-ZXM, 

2511  Jefferson  Davis  Highway,  Arlington,  Virginia  22202-3926. 

FINAL  DISPOSITION:  This  Research  Report  may  be  destroyed  when  it  is  no  longer 
needed.  Please  do  not  return  it  to  the  U.S.  Army  Research  Institute  for  the  Behavioral 
and  Social  Sciences. 

NOTE:  The  findings  in  this  Research  Report  are  not  to  be  construed  as  an  official 
Department  of  the  Army  position,  unless  so  designated  by  other  authorized  documents. 


REPORT  DOCUMENTATION  PAGE 

1.  REPORT  DATE  2.  REPORT  TYPE 

July  2008  Final 

3  DATES  COVERED  (from. . .  to) 

January  2007  -  November  2007 

4.  TITLE  AND  SUBTITLE 

Effects  of  Input  Device  and  Latency  on  Performance  While 
Training  to  Pilot  a  Simulated  Micro-Unmanned  Aerial  Vehicle 

5a  CONTRACT  OR  GRANT  NUMBER 

5b.  PROGRAM  ELEMENT  NUMBER 

622785 

6.  AUTHOR(S) 

Paula  J.  Durlach  (U.S.  Army  Research  Institute); 

Deborah  R.  Billings  (University  of  Central  Florida) 

5c.  PROJECT  NUMBER 

A790 

5d.  TASK  NUMBER 

273 

5e.  WORK  UNIT  NUMBER 

H01 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social 
Sciences 

ATTN:DAPE-ARI-IF 

12350  Research  Parkway,  Orlando,  FL,  32826-3276 

8.  PERFORMING  ORGANIZATION  REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social 
Sciences 

251 1  Jefferson  Davis  Highway 

Arlington,  VA  22202-3926 

10.  MONITOR  ACRONYM 

ARI 

11.  MONITOR  REPORT  NUMBER 

Technical  Report  1234 

12  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  is  unlimited. 

13.  SUPPLEMENTARY  NOTES 

Subject  Matter  POC:  Paula  J.  Durlach 

14  ABSTRACT  (Maximum  200  words): 

The  effects  of  input  device  and  latency  in  training  to  manually  pilot  a  simulated  micro-aerial  vehicle  (MAV)  were 
investigated.  Our  prior  research  suggested  that  performance  was  superior  when  using  a  game  controller  as 
opposed  to  a  mouse  during  missions  in  which  maneuvering  skill  and  attention  to  sensory  imagery  were  critical.  This 
experiment  investigated  whether  effects  would  persist  when  participants  were  tested  in  a  novel  environment  and 
when  some  realistic  latency  was  imposed  between  input  command  and  MAV  response.  Fifty-six  participants  were 
trained  to  operate  a  MAV  in  one  simulated  environment  and  then  tested  with  two  new  missions  in  a  novel 
environment.  Four  between-group  conditions  were  examined,  formed  by  crossing  two  2-level  factors:  input  device 
(game  controller  vs  mouse)  and  latency  period  (no  time  delay  vs.  500  ms  delay).  The  effects  of  input  device 
replicated  our  prior  research  and  also  transferred  to  the  novel  environment,  suggesting  that  input  device  rather  than 
spatial  learning  was  responsible  for  the  differences  in  performance.  No  substantial  effects  of  delay  were  found. 

15.  SUBJECT  TERMS 

Remote  control,  human-robot  interaction,  multimodal  displays  and  input  controls,  robotic  control,  teleoperation 

SECURITY  CLASSIFICATION  OF 

'  .  -.  ' 

19.  LIMITATION  OF 
ABSTRACT 

Unlimited 

20.  NUMBER 

OF  PAGES 

21  RESPONSIBLE  PERSON 

Ellen  Kinzer 

Technical  Publications  Specialist 
703-602-8047 

16  REPORT 
Unclassified 

17.  ABSTRACT 
Unclassified 

18.  THIS  PAGE 
Unclassified 

1 


11 


Technical  Report  1234 


Effects  of  Input  Device  and  Latency  on  Performance 
While  Training  to  Pilot  a  Simulated  Micro-Unmanned 

Aerial  Vehicle 


Deborah  R.  Billings 

University  of  Central  Florida 
Consortium  Research  Fellows  Program 

Paula  J.  Durlach 

U.  S.  Army  Research  Institute 


ARI-Orlando  Research  Unit 
Stephen  L.  Goldberg,  Chief 


U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
2511  Jefferson  Davis  Highway,  Arlington,  Virginia  22202-3926 


July  2008 


Amy  Project  Number _ 

622785A790  Personnel  Performance 

and  Training  Technology 


Approved  for  public  release;  distribution  is  unlimited. 


Acknowledgement 


The  authors  wish  to  acknowledge  the  Institute  for  Simulation  and  Training  in  Orlando,  FL, 
which  developed  the  simulation  environment  and  provided  technical  support  under  a  contract 
with  the  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI).  We  would 
especially  like  to  thank  Glenn  Martin  and  Michael  Whitley. 


IV 


EFFECTS  OF  INPUT  DEVICE  AND  LATENCY  ON  PERFORMANCE  WHILE  TRAINING 
TO  PILOT  A  SIMULATED  MICRO-UNMANNED  AERIAL  VEHICLE 

EXECUTIVE  SUMMARY 


Research  Requirement: 

The  introduction  and  demonstrated  usefulness  of  unmanned  aerial  vehicles  (UAVs)  has 
influenced  the  Army’s  decision  to  integrate  these  assets  into  the  Future  Combat  System  (FCS; 
Cambone,  Krieg,  Pace,  &  Wells,  2005).  The  employment  of  UAVs  has  increased  since  2001,  and 
they  are  currently  used  to  support  several  missions,  including  those  in  Iraq  and  Afghanistan 
(Cambone  et  al,  2005).  Micro-aerial  vehicles  (MAVs)  are  man-portable  UAVs,  which  lack  the 
infrastructure  requirements  of  larger  systems  (e.g.,  need  for  a  runway  and  dedicated  ground 
crew).  One  prototype  MAV  (Crane,  2005)  employs  a  ducted  fan  design,  which  gives  it  a  unique 
advantage  over  fixed  wing  aerial  vehicles — vertical  take  off  and  land,  and  the  ability  to  hover. 
This  prototype  MAV  is  slated  to  evolve  into  the  FCS’s  Class  I  UAV,  and  the  U.S.  Navy  has 
already  announced  plans  to  deploy  a  current  version  in  Iraq  to  aid  in  detection  of  improvised 
explosive  devices  (IEDs).  Due  to  the  growing  interest  in  field  uses  of  these  vehicles,  it  is 
important  that  the  MAV  man/machine  interfaces  are  designed  to  facilitate  control  and  data 
interpretation,  and  that  systematic  training  methods  and  training  standards  are  developed. 

Our  prior  research  suggested  that  a  game  controller,  affording  maneuver  control  in 
multiple  dimensions  simultaneously,  supported  superior  performance  during  MAV  operator 
training  in  simulation,  compared  with  a  mouse,  affording  control  of  only  one  dimension  at  a  time 
(Durlach,  Neumann,  &  Billings,  2008).  The  performance  measure  was  time  to  complete  the 
mission.  Moreover,  subsequent  to  training,  participants  who  had  used  the  continuous  device  (the 
game  controller)  gave  more  positive  usability  ratings  than  participants  who  had  used  the  discrete 
device  (the  mouse).  The  present  experiment  was  designed  to  further  investigate  the  mechanism 
underlying  this  difference.  The  game  controller  affords  more  focused  attention  on  the  sensory 
imagery  than  does  the  mouse,  because  it  can  be  operated  without  diverting  attention  from  the 
sensory  image.  The  mouse,  in  contrast,  requires  alternating  attention  from  the  sensory  image  to 
on-screen  controls,  which  must  be  selected  to  initiate  commands.  It  might  be  this  difference  in 
attentional  demands,  rather  than  the  continuous  vs.  discrete  control  afforded  by  the  two  devices, 
that  accounted  for  the  performance  difference  we  observed.  More  focused  attention  on  the  sensor 
imagery  may  have  allowed  users  of  the  game  controller  to  better  learn  landmarks  and  spatial 
configurations  in  the  synthetic  environment,  and  it  may  have  been  this  learning  that  supported 
better  performance.  We  reasoned  that  if  spatial  learning  were  the  cause  of  the  performance 
difference,  that  difference  should  not  be  evident  were  the  pilots  tested  in  a  novel  environment. 
Therefore,  in  the  current  research,  we  trained  pilots  in  one  environment  and  then  tested  them  in  a 
novel  environment.  In  addition,  the  possibility  that  the  effect  of  input  device  might  interact  with 
system  latency  was  investigated,  because  actual  unmanned  systems  tend  to  involve  such 
latencies. 
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Procedure: 


Four  between-group  conditions  were  examined,  formed  by  crossing  two  2-level  factors: 
input  device  (mouse  vs.  game  controller)  and  latency  period  (no  time  delay  vs.  500  ms  delay). 
Fifty-six  participants  were  randomly  assigned  to  one  of  the  four  conditions.  Participants 
completed  MAV  operator  training  in  one  simulated  environment  and  were  then  tested  in  a  novel 
environment.  Participant  performance  was  measured  in  both  the  practice/training  environment 
and  the  novel  environment.  The  primary  dependent  variable  was  completion  time  for  each 
mission,  although  we  did  measure  other  variables  such  as  collisions  and  subjective  workload. 

Findings: 

We  replicated  the  pattern  of  results  found  in  our  previous  research.  Training  missions 
were  completed  more  quickly  with  the  game  controller  than  the  mouse.  This  difference  was 
observed  in  both  the  training  and  the  test  environments.  The  effect  of  latency  failed  to  be  of 
much  consequence.  Subjective  workload  scores  were  little  affected  by  input  device  or  latency. 

Utilization  and  Dissemination  of  Findings: 

Continuous  input  devices  for  3-dimensional  navigation  of  remote  vehicles  seem  to 
support  more  efficient  mission  performance,  compared  to  discrete  input  devices.  Our  results  tend 
to  rule  out  the  explanation  of  better  spatial  learning  with  the  continuous  device  as  the  basis  for 
this  difference.  It  still  may  be  the  case  that  more  focused  visual  attention  with  our  continuous 
device  than  our  discrete  device  contributed  to  this  difference.  The  mouse  requires  the  user  to 
divide  their  attention  between  the  sensory  imagery  and  the  input  control  display;  this  division  of 
attention  may  put  cognitive  demands  on  the  operator,  which,  although  not  detected  in  subjective 
measures  of  workload,  may  nevertheless  cause  less  efficient  mission  performance. 

In  addition,  during  the  course  of  this  research,  we  have  developed  training  regimes  which 
could  be  applied  in  future  training  development  for  operators  of  MAVs.  We  have  demonstrated 
that  time  to  complete  a  mission  is  a  sensitive  performance  measure,  in  that  it  decreased  over  the 
course  of  training  and  was  sensitive  to  the  effect  of  input  device.  In  contrast,  number  of  targets 
detected  in  a  fixed-time  simulated  reconnaissance  mission  proved  to  be  insensitive  to  input 
device,  and  may  not  be  as  useful  as  a  measure  of  operator  performance. 

The  results  of  this  research  have  been  accepted  for  presentation  as  a  paper  at  the  2008 
meeting  of  the  Human  Factors  and  Ergonomics  Society.  The  results  have  been  shared  with  the 
Human-Robot  Interaction  Army  Technology  Objective  team,  led  by  the  Army  Research 
Laboratory  Human  Research  and  Engineering  Directorate.  They  have  also  been  sent  to  PM 
Unmanned  Aerial  Systems,  PM  Common  Controller,  the  Soldier  Battle  Lab,  and  the  United 
States  Army  Intelligence  Center,  Director  of  Combat  Development. 
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EFFECTS  OF  INPUT  DEVICE  AND  LATENCY  ON  PERFORMANCE  WHILE  TRAINING 
TO  PILOT  A  SIMULATED  MICRO-UNMANNED  AERIAL  VEHICLE 


Introduction 

Unmanned  aerial  vehicles  (UAVs)  are  being  created  to  serve  a  variety  of  purposes.  They 
have  many  potential  applications  including  search  and  rescue  operations  and  environmental 
monitoring  (Sanna  &  Pralio,  2005).  The  military  currently  uses  UAVs  for  reconnaissance  and 
surveillance  operations.  Cutting  edge  technology  in  sensors,  global  positioning  system  (GPS) 
receivers,  and  microelectronics  have  led  to  the  prospect  of  very  small,  lightweight,  man  portable 
micro-unmanned  aerial  vehicles  (MAVs)  that  are  deployable  almost  anywhere  (Lyon,  2004). 
These  systems  could  provide  unprecedented  situational  awareness  at  small  unit  levels;  however, 
the  successful  deployment  of  such  systems  at  the  lowest  military  echelons  will  require 
optimization  of  human-system  interaction  and  efficient  training  procedures.  Both  system 
operators  and  unit  leaders  will  require  training.  The  operators  will  require  training  on  systems 
operation  and  maintenance,  and  the  leaders  will  require  training  on  system  employment 
(Durlach,  2007).  The  military  has  recognized  the  magnitude  and  scope  of  benefits  that  remotely 
operated  systems  can  provide;  but,  their  focus  has  been  developing  the  technology,  rather  than 
optimizing  human-machine  interfaces  or  designing  training  procedures  and  standards. 


Background 

The  design  of  UAV  technology  and  controls  become  more  advanced  as  technology  is 
developed,  and  it  is  important  to  incorporate  human  factors  into  the  design  process  so  that  UAVs 
can  be  used  both  safely  and  effectively.  It  has  been  shown  that  more  accidents  resulting  in 
aircraft  damage  or  the  complete  loss  of  aircraft  have  occurred  per  flight  hour  for  UAVs  than  for 
inhabited  air  vehicles,  and  over  half  of  these  accidents  have  been  attributed  to  human  factors 
issues  (Tvaryanas,  Thompson,  &  Constable,  2005).  Hence,  human  factors  consideration  should 
be  taken  into  account  when  designing  these  man/machine  interfaces  and  controls  so  that 
operators  can  safely  and  effectively  pilot  these  unmanned  vehicles.  In  general,  there  are  two 
indispensable  human  roles  involved  in  the  operation  of  a  UAV:  a  pilot  for  the  vehicle,  and  an 
individual  responsible  for  interpreting  the  sensor  imagery  (Goodrich  et  al,  2008).  When 
controlling  a  MAV,  one  operator  may  be  required  to  fill  both  of  these  roles  simultaneously. 
While  a  substantial  body  of  research  has  been  devoted  to  the  creation  of  the  hardware  needed  for 
these  systems,  less  research  has  addressed  potential  human-computer  interface  problems  and  the 
need  for  future  training  requirements.  We  conducted  prior  research  that  addressed  the  interface 
control  display  design  of  a  simulated  MAV  and  found  several  issues  that  should  be  addressed 
when  operators  must  control  MAVs:  input  device  and  system  latency. 

In  a  previous  experiment  (Durlach,  Neumann,  &  Billings,  2008),  we  developed  a  number 
of  operator  training  scenarios,  and  used  them  to  conduct  training  of  MAV  manual  control  in  a 
simulation  environment.  In  manual  control  mode,  a  MAV  operator  can  control  maneuvers  of  the 
vehicle  in  near-real  time  (as  opposed  to  way-point  pre-programmed  navigation,  which  is 
determined  prior  to  launch).  The  sensory  image  sent  from  the  vehicle  to  the  operator  interface  is 
used  to  guide  navigation  and  avoid  obstacles.  One  purpose  of  the  experiment  was  to  determine 
the  parameters  of  performance  that  might  serve  as  suitable  measures  of  training  mastery,  and 
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thus  viable  candidates  for  establishing  training  standards.  Another  purpose  of  the  experiment 
was  to  examine  how  variation  in  operator  interface  design  would  impact  performance  during 
training.  We  found  that  time  to  complete  missions  was  sensitive  to  the  input  control  device,  such 
that  participants  provided  with  continuous  control  (a  game  controller  with  two  thumb  sticks) 
completed  missions  more  quickly  than  participants  provided  with  discrete  control  (mouse  used  to 
point  and  click  on  a  set  of  directional  icons  on  the  display  screen).  The  primary  purpose  of  the 
present  experiment  was  to  determine  if  the  temporal  performance  benefit  observed  with  the 
continuous  control  device  was  constrained  to  the  training  environment  or  whether  it  would  also 
be  observed  if  trainees  were  tested  in  a  novel  simulation  environment. 

Input  Device  Issues 

Existing  UA  Vs  are  controlled  by  a  variety  of  input  devices  including  a  touch-screen  and 
stylus  interface,  traditional  stick-and-rudder  controls,  natural  language  and  visual  gesturing 
interfaces,  and  game  controllers  (Chen,  Haas,  Pillalamarri,  &  Jacobson,  2006).  The  touch-screen 
and  stylus  interface,  as  found  on  Honeywell’s  Class  I  prototype,  allows  the  UAV  operator  to 
control  the  vehicle  directly  through  the  2-dimensional  screen.  The  more  traditional  cockpit-like 
controls,  like  those  for  the  Predator  UAV,  have  a  stick-and-rudder  design  in  order  to  more 
closely  emulate  the  real  controls  used  for  flying  manned  aircraft.  In  addition,  some  UAV  controls 
are  being  designed  that  incorporate  and  feature  game  controllers  that  function  to  maneuver  the 
UAV  as  well  as  tilt,  pan,  and  zoom  cameras  on  the  UAV.  The  Evolution  XTS,  the  Sentry®  HP, 
the  CyberBug,  and  the  Raven  RQ-1  IB  all  have  controls  similar  to  commercially  available  game 
controllers.  The  choice  of  input  device  appears  to  be  at  the  discretion  of  the  system  developer 
rather  than  on  research.  One  of  the  purposes  of  this  present  experiment  was  to  contribute  data 
upon  which  a  more  rational  choice  could  be  made. 

During  the  course  of  developing  protocols  and  standards  of  performance  for  training 
manual  control  of  a  prototype  MAV,  we  previously  investigated  the  influence  of  input  device  on 
trainee  performance,  using  game  controllers  as  opposed  to  a  2-dimensional  input  device 
(Durlach,  Neumann  &  Billings,  2008).  In  our  previous  research,  we  chose  to  use  a  mouse  as  the 
2-dimensional  input  device  in  place  of  a  touch-screen  and  stylus.  Both  methods  use  a  point-and- 
click  approach  in  which  attention  must  be  directed  away  from  the  camera  images,  but  the  mouse 
registers  inputs  more  reliably.  We  felt  this  was  especially  important  considering  that  previous 
research  found  that  users  generally  question  the  reliability  of  touch  screens  (Durlach,  Neumann 
&  Bowens,  2006).  Durlach  et  al.  (2006)  found  that  users  had  trouble  distinguishing  between 
failed  touches  and  successful  touches,  the  results  of  which  were  merely  delayed.  Other  research 
has  found  that  performance  in  pointing  tasks  was  comparable  for  a  mouse  and  a  stylus 
(MacKenzie,  Sellen  &  Buxton,  1991).  Therefore,  we  used  a  mouse  in  lieu  of  a  touch-screen  and 
stylus  so  that  input  reliability  would  not  affect  our  measures.  We  also  used  a  mouse  out  of 
convenience,  since  we  were  simulating  MAV  operation  from  a  computer  workstation.  While  this 
would  most  likely  not  be  used  in  the  field,  the  mouse  has  many  of  the  same  characteristics  of  a 
touch-screen  in  that  both  are  discrete  in  nature  and  utilize  point-and-click  methods  of  command 
input. 


We  found  that  game  controllers  have  the  advantage  of  offering  several  degrees  of 
freedom,  or  the  number  of  directions  the  object  can  be  controlled  simultaneously.  According  to 
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our  research  in  this  area,  this  form  of  input  device  may  offer  performance  benefits  over  discrete 
devices  such  as  touch  screen  and  similar  2-dimensional  input  devices  (Durlach,  Neumann  & 
Billings,  2008). 

Our  previous  research  demonstrated  that  trainees  using  game  controllers  completed 
missions  more  quickly  than  trainees  using  a  computer  mouse  (Durlach,  Neumann  &  Billings, 
2008).  It  could  be  that  a  continuous  control  device  is  more  efficient  than  a  discrete  control  device 
for  controlling  the  maneuver  of  a  vehicle  in  three  dimensions.  The  game  controller  allows  flight 
movement  in  several  dimensions  simultaneously  while  the  2-dimensional  mouse  uses  point-and- 
click  maneuvering  with  an  on-screen  control  pad  (allowing  movement  in  only  one  dimension  at  a 
time).  If  this  is  the  reason  why  we  found  superior  performance  using  the  game  controller,  the 
performance  benefit  of  using  a  continuous  input  versus  a  discrete  input  device  should  not  depend 
on  the  environment.  Consequently,  superior  performance  with  the  continuous  device  should  be 
observed  in  both  the  training  and  novel  environments. 

We  cannot  assert,  however,  that  the  only  difference  between  our  continuous  and  discrete 
input  control  conditions  was  the  “continuity”  of  the  input  device.  The  participants  using  the 
game  controller  (continuous  input  device)  could  focus  their  attention  on  the  sensor  display  more 
than  the  participants  using  the  mouse  (discrete  input  device).  Those  using  the  mouse  had  to 
divert  their  visual  attention  away  from  the  sensor  imagery  with  every  maneuver  command  they 
issued  in  order  to  ensure  that  they  were  “clicking”  on  the  proper  control  button  displayed  on  the 
screen.  Those  using  the  game  controller,  on  the  other  hand,  could  maintain  visual  attention  on  the 
sensor  imagery  (once  they  had  mastered  the  finger-to-device  mappings).  Instead  of  needing  to 
visually  locate  the  control  input,  they  could  do  it  by  feel  and  therefore  did  not  need  to  divert  their 
visual  attention  away  from  the  sensor  imagery  as  often.  It  is  possible,  therefore,  that  participants 
using  the  game  controller  had  greater  opportunity  to  learn  about  the  spatial  characteristics  of  the 
environment  compared  with  the  participants  using  the  mouse.  This  in  turn  may  have  allowed 
participants  using  the  game  controller  to  better  anticipate  obstacles  in  the  environment,  maneuver 
more  smoothly,  and  complete  missions  more  rapidly.  To  the  extent  that  differential  knowledge  of 
the  training  environment  accounted  for  the  performance  differences  observed  across  input 
conditions,  this  performance  difference  should  fail  to  be  manifest  in  a  novel  environment.  The 
previous  differential  in  spatial  learning  would  be  of  no  benefit  in  a  new  environment. 

Input  devices  should  enhance  training  so  that  attention  is  predominately  on  camera 
imagery  and  other  important  aspects  of  manually  piloting  the  MAV  rather  than  the  actual  control 
devices,  and  the  game  controller  appeared  to  offer  this  advantage.  In  addition,  performance 
tended  to  be  better  overall  when  the  control  method  matched  the  function  of  the  input  command. 
For  example,  a  discrete  command  (such  as  hovering  or  taking  a  photograph)  is  probably  best 
implemented  as  a  single  discrete  input  (such  as  a  mouse  click  or  pressing  a  button  on  the  game 
controller).  Conversely,  a  continuous  command  (such  as  movement  through  a  3-D  space)  is 
probably  best  implemented  as  continuous  commands  (e.g.,  the  physical  movement  of  thumbstick 
controls  that  directly  translates  into  MAV  movement;  Durlach,  Neumann,  &  Billings,  2008).  The 
game  controller  offered  both  continuous  control  and  discrete  control. 

The  purpose  of  our  current  experiment  was  to  address  which  factor  was  responsible  for 
the  differences  in  performance:  the  input  device  itself  or  spatial  learning.  We,  therefore, 
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examined  whether  the  effects  of  input  device  would  persist  across  a  change  in  environment.  If 
spatial  learning  were  responsible  for  the  differences  in  performance,  those  differences  should  be 
specific  to  the  training  environment;  consequently,  the  differences  would  not  appear  when 
participants  are  placed  in  a  new  simulated  environment.  If  the  differences  persist  in  the  novel 
environment,  we  can  eliminate  differential  spatial  learning  as  the  mechanism  underlying  the 
performance  effects. 


Latency  Issues 

Latency  refers  to  delays  inherent  in  a  system.  Latencies  are  caused  primarily  by 
transmission  time  requirements,  broken  signals,  or  time  required  for  computation.  Latency  can 
reduce  operator  performance  in  general,  cause  errors  in  operator  judgment,  and  decrease 
performance  in  manual  control  and  tracking  tasks  (Ferrell,  1965);  however,  much  of  the  research 
on  the  effects  of  latency  has  provided  inconsistent  results.  Some  research  has  shown  that 
unmanned  vehicle  operators  experiencing  a  delay  of  greater  than  1 00  ms  may  have  increased 
performance  errors  because  they  cannot  correct  their  mistakes  quickly  and  effectively;  these 
operators  will  reduce  the  speed  of  the  UA  V  in  order  to  maintain  control  of  the  vehicle  (Miall  & 
Jackson,  2006).  Other  studies  have  shown  that  target  acquisition  and  tracking  performance 
declined  significantly  when  introduced  with  latencies  anywhere  from  approximately  200  ms  to 
about  320  ms,  while  other  researchers  found  decrements  at  500  ms  (MacKenzie  &  Ware,  1993; 
Lane  et  al,  2002).  Further  research  demonstrated  that  operator  performance  degraded 
significantly  only  when  system  latencies  were  greater  than  1 .5  seconds  (Lane  et  al,  2002;  Chen, 
Haas,  &  Barnes,  2007).  These  various  findings  do  not  allow  one  to  predict  whether  a  specific 
latency  will  or  will  not  cause  significant  performance  issues. 

In  our  previous  research,  we  did  not  impose  any  degree  of  latency  on  the  system 
(Durlach,  Neumann,  &  Billings,  2008).  The  current  research  seeks  to  examine,  if  the  superiority 
of  the  game  controller  as  an  input  device  still  holds,  when  there  is  a  latency  of  500  ms  imposed 
between  input  and  MAV  response.  We  chose  a  500  ms  latency  because  subject  matter  experts 
suggested  500  ms  was  a  realistic  period  of  latency  in  MAV  operation.  The  latency  of  500  ms  has 
been  shown  to  produce  drastic  performance  decrements  in  some  studies;  but,  minimal  effects 
according  to  other  research  (MacKenzie  &  Ware,  1993;  Lane  et  al,  2002). 

Current  Research 

The  current  research  is  intended  to  replicate  and  validate  the  findings  of  our  previous 
research  when  the  realistic  limitations  of  latency  and  novel  environments  are  applied  to  the 
simulation  (Durlach,  Neumann  &  Billings,  2008).  This  experiment  used  a  2x2  between-groups 
design  with  input  device  type  (mouse  vs.  game  controller)  and  latency  between  operator  input 
and  MAV  movement  (0  ms  vs.  500  ms  delay)  being  the  independent  variables.  Each  participant 
was  randomly  assigned  to  one  of  four  conditions.  Participants  completed  several  training 
missions  in  one  simulated  environment,  and  then  they  completed  two  additional  missions  in  a 
novel  simulated  environment,  using  the  same  input  device  and  latency  period  as  they  used  in  the 
previous  missions.  All  the  missions  used  in  the  training  environment  were  the  same  as  those  used 
in  our  previous  research.  The  dependent  variable  that  we  were  most  interested  in  was  completion 
times  for  the  timed  missions.  Two  of  the  missions  in  this  study  were  fixed-time  reconnaissance 
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missions;  for  these,  the  dependent  variable  of  interest  was  the  number  of  targets  photographed.  In 
addition,  we  collected  collision  data,  subjective  workload  assessments,  and  perceived  latency 
assessments  for  all  of  the  missions. 


Method 


Participants 

Twenty-eight  male  and  twenty-eight  female  participants  from  the  University  of  Central 
Florida  area  completed  this  experiment  in  exchange  for  monetary  compensation  or  college 
course  credit.  Ten  other  participants  (all  female)  failed  to  meet  the  initial  training  criteria,  and 
these  participants  were  excused  from  the  study.  All  participants  were  at  least  18  years  old,  and 
stated  they  had  normal  color  vision  and  vision  correctable  to  20/20.  Of  the  participants  who 
completed  the  entire  experiment,  the  mean  age  was  2 1 .4  years  old.  Each  participant  signed  an 
informed  consent  form  before  any  testing  began. 

Materials 

A  combination  of  subjective  measures  and  questionnaires  were  administered  during  this 
experiment,  several  of  which  were  given  on  a  laptop  computer.  The  paper-based  materials 
included  the  Hidden  Patterns  Test,  a  matching  worksheet,  two  scaled  maps  of  the  simulated 
environments  used  in  the  missions,  and  a  modified  version  of  the  Cooper-Harper  Handling 
Qualities  Rating  Scheme  (ETS,  1976;  Cummings,  Myers,  &  Scott,  2006). 

Research  has  demonstrated  that  spatial  ability,  as  measured  by  existing  psychometric 
tests,  is  significantly  associated  with  the  ability  to  learn  spatial  information  from  a  desktop 
virtual  environment  (Waller,  Knapp,  &  Hunt,  2001).  In  our  experiment  we  used  the  Hidden 
Patterns  test,  which  measures  the  flexibility  of  closure,  to  address  spatial  ability.  More 
specifically,  this  test  examines  selection  perception,  or  the  ability  of  a  participant  to  search  for 
and  recognize  a  particular  visual  pattern  among  other  distracting  visual  stimuli  (Boehm-Davis, 
Holt  &  Hansberger,  1997).  We  used  the  Hidden  Patterns  test  because  in  our  previous  research  it 
was  significantly  correlated  with  more  of  the  critical  dependent  performance  measures  than  other 
spatial  tests  that  were  used  (Durlach,  Neumann,  &  Billings,  2008). 

The  matching  worksheet  presented  an  image  of  the  operator  control  unit  (OCU)  display, 
and  the  participant  was  instructed  to  match  different  components  on  the  OCU  display  to  their 
functions  in  order  to  assess  participant  understanding  of  the  system.  The  scaled  maps  of  the 
terrain  databases  (Fort  Polk  and  Fort  McKenna)  were  given  to  participants  so  that  they  could 
mark  recalled  target  locations  detected  during  tactical  missions.  The  Cooper-Harper  Handling 
Qualities  Rating  Scheme,  developed  in  1 969,  was  designed  to  subjectively  assess  pilot  opinions 
regarding  some  aspect  of  a  specific  flying  task  (Best,  &  Schopper,  1 995).  The  Cooper-Harper 
rating  scale  consists  of  10  possible  ratings,  with  1  signifying  excellent  or  highly  desirable 
characteristics  and  10  representing  major  deficiencies.  The  modified  version  of  this  rating  scale 
(unique  to  this  experiment)  asked  participants  to  rate  their  opinions  regarding  operating  system 
delay  and  perceived  aircraft  handling  performance  and  consisted  of  only  7  possible  ratings.  The 
scale  is  illustrated  in  Table  1. 
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Table  1.  Modified  Cooper-Harper  Handling  Qualities  Rating  Scheme,  Unique  to  our  Experiment 


System 

Opinion 

Rating 

Excellent 

I  had  no  problems  with  system  delays;  I  didn’t  notice  any  delays 

1 

Good 

I  noticed  some  delays;  but  they  didn’t  affect  how  I  piloted  the  MAV. 

2 

Fair,  some  room  for 
improvement 

I  noticed  some  delays,  which  affected  how  I  piloted  the  MAV;  but  I 
was  able  to  cope  with  them  easily,  and  they  didn’t  really  bother  me. 

3 

Moderate 

deficiencies,  could  be 
improved 

The  delays  were  somewhat  annoying,  and  I  was  aware  of  having  to 
compensate  for  them  in  piloting  the  MAV. 

4 

Objectionable 
deficiencies,  should 
be  improved 

The  delays  were  very  annoying;  they  really  affected  how  I  controlled 
the  MAV;  but  I  don’t  think  my  mission  performance  suffered  as  a 
result. 

5 

Major  deficiencies, 
must  be  improved 

The  delays  were  very  annoying;  they  really  affected  how  I  controlled 
the  MAV.  I  think  my  mission  performance  suffered  as  a  result. 

6 

Current  system 
unacceptable 

The  delays  were  unacceptable.  Because  of  system  delays,  there  were 
times  when  I  was  unable  to  maintain  control  of  the  MAV. 

7 

The  NASA  TLX  was  administered  via  a  laptop  computer,  and  subjective  workload  scores 
were  automatically  calculated  (Hart  &  Staveland,  1988).  A  demographics  survey  and  a  usability 
questionnaire  were  also  given  on  a  laptop  computer.  The  demographics  survey  consisted  of  15 
items  that  asked  participants  primarily  about  computer  and  video  game  experience  and  usage. 

The  usability  survey  required  the  participants  to  rate  32  different  items  on  a  10  point  Likert  scale, 
regarding  participant  experiences  with  the  OCU  system. 

Apparatus 

The  experiment  was  conducted  using  two  networked  computers  and  one  non-networked 
laptop.  The  OCU,  the  MAV  simulator,  and  the  synthetic  terrain  database  were  all  run  on  the  first 
computer  (which  the  participants  used  to  pilot  the  MAV).  Participants  sat  at  a  desk  and 
interacted  with  the  OCU  using  either  a  standard  two-button,  one-wheel  Dell  optical  mouse  or  a 
Logitech  dual-thumbstick  game  controller.  The  One  Semi-Automated  Forces  (ONESAF) 

Testbed  Baseline  (OTB  version  2.5)  was  loaded  on  the  second  computer.  This  allowed  the 
experimenter  to  introduce  dismounted  Soldiers  and  ground  vehicles  into  the  terrain  database.  The 
non-networked  laptop  was  used  to  administer  the  NASA  TLX  workload  questionnaire  as  well  as 
a  demographics  survey  and  a  usability  questionnaire. 

The  Simulated  MA  V 

The  simulated  MAV  used  in  the  experiment  was  based  loosely  on  the  Class  I  t-MAV 
prototype,  developed  by  the  Defense  Analysis  Research  Project  Agency’s  MAV  Advanced 
Technology  Demonstration  (See  Figure  1).  The  MAV  incorporated  a  ducted  fan  design  that 
allowed  for  vertical  lift-off  and  landing,  hovering,  rotating  in  place,  and  speeds  of  up  to  6  knots 
(in  manual  mode).  In  addition,  the  MAV  was  equipped  with  two  fixed  cameras:  one  was  pointed 
straight  down,  and  one  was  pointed  in  a  forward  direction.  MAV  flight  included  inertial 
properties.  For  example,  when  a  hover  command  was  issued  while  the  MAV  was  in  motion,  it 
slowed  gradually  before  coming  to  a  complete  stationary  hover.  In  addition,  forward  movement 
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of  the  MAV  caused  the  MAV  (and  its  fixed)  cameras  to  tilt  by  one  degree  for  every  knot  of 
forward  speed  (like  the  actual  t-MAV  prototype). 


Figure  1.  The  2005  t-MAV  prototype  by  Honeywell. 


Operator  Control  Unit  (OCU) 

The  OCU  (see  Figure  2)  showed  information  necessary  for  mission  completion  (See 
Appendix  B  for  mapping  of  commands  to  the  OCU  interface).  The  OCU  display  consisted  of 
video  sensor  imagery  (two  camera  views,  but  only  one  camera  could  be  viewed  at  a  time),  an 
overhead  map  view  of  the  terrain  database  (satellite  view),  a  discrete  control  pad  used  for  issuing 
flight  commands  (See  Figure3),  an  altimeter,  a  heading  tape,  and  several  task  bar  icons  used  for 
issuing  additional  MAV  commands  (e.g.,  take-off,  switching  camera  views,  etc.).  The  current 
position  of  the  MAV  was  always  displayed  on  the  map  view.  The  mission  timer  was  located  in 
the  upper  right  of  the  screen.  Timing  for  each  of  the  exercises  was  initiated  by  the  operator, 
beginning  with  the  issue  of  the  take-off  command  and  ending  when  the  MAV  was  grounded. 
After  the  operator  gave  the  “take-off’  command,  the  MAV  rose  to  an  altitude  of  60  feet  above 
the  ground,  and  any  maneuver  commands  given  during  this  time  had  no  effect.  After  the  “take¬ 
off’  altitude  was  achieved,  a  red  stop  icon  on  the  control  unit  illuminated;  only  after  this  could 
participants  give  maneuver  commands. 
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Figure  2,  The  OCU. 
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Figure  3.  Illustration  of  the  discrete  input  control  display. 


Input  Devices 

Participants  used  one  of  two  input  devices  during  the  experiment:  a  mouse  or  a  game 
controller.  The  mouse  was  a  2 -button/1  -wheel  Dell  optical  mouse  with  a  USB  connector  that  was 
placed  on  a  mouse  pad.  The  game  controller  was  a  Logitech  dual-thumbstick  controller  with  a 
USB  connector  (See  Figure  4).  Both  devices  enabled  the  participant  to  issue  the  same  commands 
to  the  MAV,  only  in  different  ways.  Participants  using  the  mouse  simply  had  to  click  on  the 
desired  function  on  the  screen  to  activate  the  OCU  (See  control  pad  in  Figure  3).  Conversely, 
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participants  using  the  game  controller  were  required  to  issue  commands  to  the  OCU  via  the 
buttons  on  the  game  controller  itself.  The  same  visual  feedback  was  given  on  the  on-screen 
control  pad  (i.e.,  the  command  key  lit  up),  regardless  of  what  type  of  input  device  was  used. 


Scroll 

through 

icons 


Forward 


Press  to  take  off  and  land 


Activate 

Camera 

selection 

Icons 

(release 

to  select) 


Reverse 


Down 


Figure  4.  Illustration  of  the  game  controller  with  functions  labeled.  Left-right/lateral  movements 
(left  thumbstick)  did  not  change  vehicle  heading. 

Synthetic  Environments 


During  the  first  part  of  the  experiment,  participants  trained  to  pilot  the  MAV  in  a 
simulated  environment  based  on  an  area  of  Fort  Polk,  LA.  They  were  subsequently  required  to 
perform  two  missions  in  a  novel  synthetic  environment  based  on  an  area  at  Fort  Benning,  GA. 
Ficmres  5  and  6  show  overhead  views  of  each  of  these  environments.  Besides  have  different 


Figure  5.  Overhead  map  view  of  the  simulated  Fort  Polk  training  environment. 
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Figure  6.  Overhead  map  view  of  the  simulated  Fort  Benning  transfer  environment. 

road  and  building  configurations,  the  two  areas  differed  in  the  placements  of  trees,  powerlines, 
and  other  obstacles  (which  are  not  obvious  in  the  figures). 

Procedure 

At  the  start,  participants  signed  a  voluntary  informed  consent  form.  Then  they  were 
instructed  to  complete  a  demographics  survey,  and  participants  were  subsequently  given  the 
Hidden  Patterns  test  to  measure  spatial  ability.  Following  these  preliminaries,  participants  were 
assigned  to  one  of  four  conditions,  determined  by  a  2  x  2  between-groups  design  where  the 
factors  were  input  device  (mouse  or  game  controller)  and  latency  (no  delay  or  500  ms  delay). 
There  were  an  equal  number  of  male  and  female  participants  in  each  condition.  Each  participant 
was  then  trained  to  fly  a  simulated  MAV,  equipped  with  a  printed  operator  manual  as  well  as 
experimenter  assistance.  After  participants  were  trained  on  MAV  capabilities  and  control 
functions,  they  were  then  required  to  successfully  complete  six  practice  exercises  within  an 
allotted  time,  all  of  which  in  the  Fort  Polk  environment. 

The  first  four  practice  exercises  focused  on  allowing  participants  to  learn  the  mapping  of 
all  buttons  and  their  respective  commands.  For  example,  one  practice  exercise  required 
participants  to  take  several  pictures  from  the  airborne  MAV,  and  another  practice  exercise 
required  participants  to  perform  maneuvers  with  the  MAV  as  commands  were  given  orally  by 
the  experimenter.  The  last  two  practice  exercises  incorporated  several  pre-determined  mission 
parameters  and  required  movement  over  greater  distances  for  successful  MAV  operation.  See 
Table  2  for  descriptions  of  all  practice  exercises.  Also  see  Figures  7-8  for  the  routes  that 
participants  were  instructed  to  follow  in  the  exercises.  Participants  who  were  unable  to  meet  the 
criterion  times  on  these  practice  exercises  after  5  attempts  were  dismissed  from  the  experiment. 
Participants  who  successfully  completed  the  practice  exercises  within  the  criterion  times 
continued  on  with  the  experiment. 
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Table  2.  Practice/Training  Exercise  Instructions.  All  Exercises  take  Place  in  the  Simulated 
Environment  of  Fort  Polk 


Exercise 

Instructions 

Practice  Exercise 
1  (read) 

Execute  the  Take-off  command.  When  the  Red  Stop  icon  illuminates  (showing  that 
take-off  altitude  has  been  reached),  execute  the  Land  command.  This  must  be 
completed  in  30  seconds  (:30)  or  less. 

Practice  Exercise 
2  (read) 

Execute  the  Take-off  command.  At  or  before  the  completion  of  take-off,  activate  the 
view  window  for  camera  2  (downward  view).  Take  a  snapshot  with  camera  2. 
Activate  the  view  window  for  camera  1  (forward  view).  Take  a  snapshot  with  camera 

1 .  Execute  the  Land  command.  This  must  be  completed  in  40  seconds  ( :40)  or  less. 

Practice  Exercise 

3  (read) 

The  upper  altitude  alarm  will  be  set  at  150  feet  and  activated.  Execute  the  Take-off 
command.  Ascend  to  150  feet  and  trigger  the  alarm  (you’ll  hear  a  warning  sound). 
Immediately  descend  to  50  feet  or  below  without  hitting  the  ground.  Ascend  back  up 
to  100  feet  but  less  than  150  feet.  Rotate  the  MAV  360-degrees  without  dropping 
below  100  feet.  It  is  reauired  that  the  heading  tape  shows  the  number  “0”  after 
completing  1  rotation  with  the  MAV.  The  “0”  must  remain  in  the  forward  camera 
view  window  before  landing.  Execute  the  Land  command.  This  exercise  must  be 
completed  in  1  minute  35  seconds  (1:35)  or  less. 

Practice  Exercise 
4A  (auditory 
commands) 

Practice  Exercise 
4B  (auditory 
commands) 

For  this  exercise,  you  will  follow  a  series  of  oral  commands  issued  by  the 
experimenter.  After  take-off  and  as  soon  as  the  Red  Stop  icon  illuminates,  you  will 
immediately  begin  to  hear  a  series  of  flight  commands.  Commands  will  be  given  as 
fast  as  you  can  correctly  comply.  Once  the  correct  feedback  is  observed  from  the 

OCU,  the  experimenter  will  proceed  to  the  next  command. 

Practice  4A:  The  first  series  of  commands  after  take-off  should  be:  Ascend .  descend, 
fly  forward,  (strafe)  right,  (strafe)  left,  fly  backwards,  rotate  right,  rotate  left,  land. 

This  exercise  must  be  completed  in  1  minute  5  seconds  (1 :05)  or  less. 

Practice  4B:  The  second  series  of  commands  after  take-off  should  be:  flv  forward, 
rotate  left,  fly  backwards,  rotate  right,  (strafe)  right,  (strafe)  left,  activate  camera  2, 
take  snapshot  with  camera  2,  ascend,  descend,  activate  camera  1,  take  snapshot  with 
camera  1,  fly  forward,  land. 

This  exercise  must  be  completed  in  1  minute  25  seconds  (1 :25)  or  less. 

Practice  Exercise 

5  (racetrack) 

The  experimenter  will  load  and  run  this  mission  autonomously  and  will  point  out  the 
Landing  Zone  (LZ)  on  the  (H)  building.  After  the  autonomous  mission  finishes,  the 
simulation  will  be  reset.  You  must  now  manually  pilot  the  MAV  around  the  gray 
pathway  while  remaining  to  the  left  of  the  four  red  poles  and  then  land  in  the  correct 
LZ.  If  you  do  not  follow  directions  or  if  you  crash,  you  will  be  required  to  restart  this 
exercise  from  the  beginning.  This  exercise  must  be  completed  in  3  minutes  50 
seconds  (3:50)  or  less. 

11 


After  the  practice  exercises,  participants  were  required  to  complete  five  missions,  and 
experimenter  interaction  with  participants  during  this  portion  of  the  study  was  extremely  limited. 
The  first  three  missions  were  performed  in  the  Fort  Polk  environment  (See  Table  3).  Missions  1 
and  2  were  similar  to  the  last  two  practice  exercises,  but  had  to  be  completed  unaided  by  the 
display  of  waypoints  on  the  situation  awareness  map.  Mission  1  involved  piloting  the  MAV 
around  a  track  that  was  marked  by  poles  in  the  synthetic  environment.  Mission  2  involved 
navigating  the  MAV  through  a  slalom  course.  For  these  missions,  participants  who  committed  a 
collision  were  immediately  stopped  and  required  to  restart  the  mission. 


Table  3.  Instructions  for  Missions  1,  2  &  3,  Which  occurred  in  Fort  Polk  Environment;  these  are 
Identical  to  Missions  1  through  3  in  our  Previous  Research _ 


Fort  Polk 
Mission 

Instructions 

Mission  1 

This  mission  is  a  repeat  of  practice  exercise  #5,  where  you  piloted  the  MAV  around  the  gray 
pathway  while  remaining  to  the  left  of  the  four  red  poles.  The  difference  in  this  mission  is  that 
there  will  be  no  waypoints  on  the  satellite  view  map.  You  need  to  try  to  complete  this  mission 
as  auickly  as  possible  but  also  without  any  collisions  -if  you  have  a  collision  or  if  you  deviate 

from  the  course,  you  will  be  required  to  redo  the  mission  from  the  beginning  until  you  complete 

it  without  a  collision. 

You  will  manually  pilot  the  MAV  around  the  gray  pathway  while  remaining  to  the  left  of  the 
four  red  poles,  and  then  land  on  the  (H)  building.  When  ready,  press  OK  to  start  the  timer. 

Execute  the  Take-off  command.  Complete  one  lap  around  the  four  red  poles  and  stay  over  the 
gray  path.  Land  on  the  (H)  building. 

Mission  2 

This  mission  repeats  practice  exercise  #6,  where  you  navigated  a  series  of  red  and  green  poles. 

You  will  also  take  two  snapshots  of  the  C2  vehicle  at  the  end  of  the  run.  The  difference  in  this 
mission  is  that  there  will  be  no  waypoints  on  the  satellite  view  map.  Complete  the  mission  by 
flying  through  the  series  of  red  and  green  poles,  and  then  return  to  your  start  point  to  take  the 
snapshots  of  the  C2  vehicle.  You  need  to  try  to  complete  this  mission  as  quickly  as  possible  but 
also  without  any  collisions  -if  you  have  a  collision  or  if  you  deviate  from  the  course,  you  will  be 

required  to  redo  the  mission  from  the  beginning  until  you  complete  it  without  a  collision. 

You  must  complete  the  obstacle  course  manually.  After  you  finish  navigating  around  the  poles 
(right  of  the  green  poles  and  left  of  the  red  poles),  you  will  need  to  take  snapshots  of  the  C2 
vehicle  with  both  cameras.  When  ready,  press  OK  and  then  execute  Take-off.  Complete  the 
obstacle  course.  Take  snapshot  of  the  C2  vehicle  with  camera  1.  Take  snapshot  of  the  C2  vehicle 
with  camera  2.  Land,  but  do  NOT  land  on  the  C2  vehicle. 

Mission  3 

This  mission  involves  using  the  MAV  to  do  reconnaissance  work.  You  will  get  a  handout  titled 
“Mission  3  Intel  &  Recon.”  Review  this  with  the  experimenter,  and  then  complete  the  required 
tasks.  This  is  primarily  a  target  identification  mission.  The  experimenter  may  ask  you  for 
situational  updates  during  this  mission. 

Review  the  “Mission  3  Intel  &  Recon”  handout.  The  experimenter  will  load  the  mission  files  and 
scenario.  You  will  have  a  limited  time  to  identify  as  many  targets  as  possible  on  the  map. 

Positive  ID  can  only  be  achieved  by  taking  snapshots  of  each  entity  with  both  the  forw  ard  and 
downward  cameras,  and  each  entity  must  be  centered  in  the  frame  so  that  the  center  ( )  overlay  is 
touching  part  of  the  entity.  When  ready  press  OK  to  begin  the  mission  and  start  the  timer. 
Immediately  begin  looking  for  entities  to  identify  via  the  cameras.  The  experimenter  will  tell  you 
when  time  has  expired. 
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Mission  3  was  more  tactical  in  that  participants  were  given  an  Intel  sheet  depicting 
pictures  of  various  targets,  and  they  were  told  to  photograph  as  many  entities  as  possible  with 
both  the  forward  camera  and  the  downward  camera.  Fourteen  targets  (Soldiers  and  vehicles) 
were  situated  in  the  synthetic  terrain  for  this  mission.  Participants  were  allowed  free  flight  in  the 
environment;  but  after  7  minutes  were  instructed  to  return  to  the  launch  site  and  land.  They  were 
subsequently  asked  to  mark  where  they  detected  targets  on  a  paper  map. 


Figure  7.  Mission  1 :  Fort  Polk/practice  environment  racetrack  route. 


Figure  8.  Mission  2:  Fort  Polk/practice  environment  slalom  route. 
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The  final  two  missions  (outlined  in  Table  4)  were  completed  in  the  new  synthetic 
environment  (Fort  Benning).  Mission  4  involved  piloting  the  MAV  through  a  designated  course 
(similar  to  Missions  1  and  2).  See  Figure  9  for  a  map  of  the  route  that  was  taken  in  Mission  4. 
Mission  5  was  similar  to  Mission  3  in  that  participants  were  given  an  Intel  sheet  with  various 
targets  and  asked  to  fly  through  the  simulated  environment  and  photograph  the  targets.  In  this 
case,  however,  all  the  targets  were  dismounted  Soldiers,  with  some  located  on  roof  tops  or 
windows,  and  participants  were  instructed  to  photograph  the  face  of  each  individual.  After  7 
minutes,  participants  were  instructed  to  return  to  the  launch  point  and  land. 


Table  4.  Instructions  for  Missions  4  and  5,  Which  took  Place  in  an  Area  of  Fort  Benning 


Fort  Benning 
Mission 

Instructions 

Mission  4 

This  mission  involves  using  the  MAV  to  navigate  around  obstacles.  Complete  the 
mission  by  piloting  the  MAV  around  the  roadway,  flying  through  the  series  of  red  and 
green  poles,  and  then  landing  in  the  designated  area.  You  must  fly  to  the  left  of  the 
red  poles  and  to  the  right  of  the  green  poles.  Also,  be  sure  to  stay  close  to  the 
pathway!  You  need  to  try  to  complete  this  mission  as  quickly  as  possible  but  also 
without  any  collisions  or  deviations  from  the  course  -if  you  have  a  collision,  you  will  be 

reauired  to  redo  the  mission  from  the  beginning  until  you  complete  it  without  a 

collision. 

The  experimenter  will  load  and  run  this  mission  autonomously  with  waypoints  visible. 
Observe  how  the  MAV  passes  to  the  right  of  all  green  poles  and  to  the  left  of  all  red 
poles.  You  must  now  complete  the  course  manually.  You  must  navigate  the  obstacle 
course  without  crashing  before  you  can  move  on  to  the  final  mission.  When  ready,  press 
OK  and  then  execute  Take-off.  Complete  the  obstacle  course.  Land  in  the  designated 
area.  This  exercise  must  be  completed  without  crashing  in  as  many  tries  as  necessary. 
There  is  no  time  limit. 

Mission  5 

This  mission  involves  using  the  MAV  to  do  more  reconnaissance  work.  You  will  get  a 
handout  titled  “Mission  5  Intel  &  Recon.”  Review  this  with  the  experimenter,  and  then 
complete  the  required  tasks.  This  is  primarily  a  target  identification  mission.  The 
experimenter  may  ask  you  for  situational  updates  during  this  mission. 

Review  the  “Mission  5  Intel  &  Recon”  handout.  The  experimenter  will  load  the  mission 
files  and  scenario.  You  will  have  a  limited  time  to  identify  as  many  targets  as  possible 
on  the  map.  Positive  ID  can  only  be  achieved  by  taking  snapshots  of  each  entity  with  the 
forward  camera  so  that  the  person’s  face  is  visible  and  identifiable.  When  readv  press 

OK  to  begin  the  mission  and  start  the  timer.  Immediately  begin  looking  for  entities  to 
identify  via  the  cameras.  The  experimenter  will  tell  you  when  time  has  expired. 
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Figure  9.  Mission  4:  Fort  Benning/novel  environment  transfer  skills  mission  route. 


After  each  mission,  participants  completed  a  modified  Cooper-Harper  Latency  Rating 
Scale  along  with  the  NASA  TLX.  In  addition,  after  Missions  3  and  5,  participants  were  given  a 
scaled  map  view  of  the  terrain  database  and  asked  to  mark  the  locations  of  the  targets  that  they 
identified.  After  Mission  5,  participants  were  given  a  usability  questionnaire,  which  asked 
questions  in  regard  to  the  system  and  its  piloting  capabilities.  Finally,  participants  were  given  a 
debriefing  form  and  compensated  for  their  time. 


Results  &  Discussion 


Data  Analysis 

Before  any  analyses  were  performed,  data  was  screened  for  any  issues  that  could 
potentially  affect  the  results.  Outliers  beyond  ±  2  standard  deviations  (SD)  were  transformed  to  1 
unit  (based  on  each  dependent  measure)  outside  the  most  extreme  positive  or  negative  score 
falling  within  2  SD  of  the  mean  (Tabachnick  &  Fidell,  1996).  Scores  on  the  Hidden  Patterns 
spatial  test  were  converted  to  standardized  scores  for  the  analyses.  In  addition,  the  responses  to 
the  three  demographics  questions  related  to  video  game  experience  and  skills  were  combined 
into  a  single  index,  video  game  experience  (VGE).  VGE  was  correlated  significantly  with  gender 
(Spearman  r  -  .36),  with  higher  VGE  associated  with  male  participants.  The  four  conditions 
were  checked  for  equivalence  on  VGE  and  spatial  ability;  no  significant  differences  across 
conditions  were  detected. 
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We  intended  to  use  analysis  of  covariance  (ANCOVA)  with  spatial  ability  scores 
(standardized  Hidden  Pattern  scores)  and  VGE  as  covariates.  Before  any  data  was  analyzed,  data 
were  checked  to  make  sure  ANCOVA  assumptions  were  met  for  normality  and  parallelism  of 
covariates;  if  not,  then  non-parametric  tests  were  conducted  (Mann- Whitney  U). 

Practice  Exercises 

Table  5  shows  the  distribution  of  the  number  of  attempts  required  for  successful 
completion  of  each  practice  exercise.  Both  spatial  ability  and  VGE  tended  to  correlate  negatively 
with  number  of  attempts  required  to  complete  the  practice  exercises.  The  only  significant 
correlation,  however,  was  between  spatial  ability  and  number  of  attempts  required  to  complete 
the  third  practice  exercise  (Spearman  r  =  -.36).  For  the  racetrack  practice  exercise,  the  number  of 
attempts  violated  the  ANCOVA  assumption  of  normality.  The  Mann-Whitney  U  test  indicated 
that  the  number  of  attempts  was  significantly  affected  by  input  device,  U  =  261.5 ,p  =  .03.  The 
mean  number  attempts  for  the  mouse  was  2.5  ( SD  =  1.4),  and  the  mean  number  of  attempts  for 
the  game  controller  was  1.8  (SD  =1.1).  For  the  slalom  practice  exercise,  no  significant  effects  of 
input  device  or  delay  on  number  of  attempts  were  found. 

Table  5.  Number  of  Participants  Requiring  1  to  5  Attempts  to  Complete  each  Practice  Exerci se 


Attempts:  Practice 

Exercise 

Criterion  Time 
(Minutes:  Seconds) 

1 

2 

3 

4 

5 

1  (written) 

0:30 

55 

1 

0 

0 

0 

2  (written) 

0:40 

56 

0 

0 

0 

0 

3  (written) 

1:35 

31 

22 

2 

1 

0 

4a(oral) 

1:05 

55 

1 

0 

IT 

ro- 

4b  (oral) 

1:25 

56 

0 

0 

0 

Racetrack 

3:50 

26 

9 

11 

7 

3 

Slalom 

5:00 

25 

18 

7 

3 

3 

Note.  For  exercises  1,  2,  and  3,  participants  read  a  list  of  maneuvers  themselves.  For  exercises  4a 
and  4b,  participants  responded  to  a  list  of  maneuvers  read  aloud  by  the  experimenter.  The  final 
two  practice  exercises  involved  negotiating  courses.  N  =  56. 


A  2  x  2  (input  device  x  latency)  between  subjects  ANCOVA  was  performed  on  workload 
scores  following  participant  completion  of  the  practice  exercises.  Spatial  ability  was  found  to  be 
a  significant  covariate  of  subjective  workload  scores,  F(l,50)  =  4.93,  p  =  .03,  rip  =  .09.  VGE 
was  also  found  to  be  a  significant  covariate,  F(l,50)  =  l.\6,p  =  .01,  rip2  =  .13.  Both  higher 
spatial  ability  and  higher  VGE  were  associated  with  lower  subjective  workload  (Spearman  r’s  - 
-.25  and  -.37  for  spatial  ability  and  VGE  respectively) .  The  subjective  workload  scores  for  the 
practice  exercises  were  also  significantly  affected  by  input  device,  F(\,  50)  =  5.12 ,p  =  .03,  r|P2  = 
.09.  Workload  scores  were  significantly  higher  for  participants  using  the  mouse  (mean  59.8;  SD 
=  18.5)  than  for  participants  using  the  game  controller  (mean  48.1;  SD  =  20.7).  This  difference 
was  not  apparent  in  the  baseline  TLX  measure,  /*’(  1 ,50)  =  0.47  ,p  =  .50.  Mean  baseline  TLX 
scores  were  29.3  (SD  =  17)  and  30.9  (SD  =  18.5)  for  the  mouse  and  game  controller  conditions, 
respectively. 
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The  ANCOVA  was  performed  on  latency  ratings  given  following  completion  of  the 
practice  excises.  Ratings  failed  to  be  significantly  related  to  VGE  or  spatial  ability,  but  were 
significantly  affected  by  input  device,  F(l,  50)  =  4.91  ,/?  =  .03.  Mean  latency  ratings  were  3.0 
(SD  =  1 .3)  and  2.3  (SD  =  1 .2)  for  the  mouse  and  the  game  controller,  respectively.  These  ratings 
failed  to  be  directly  affected  by  latency  itself  (F  <  1).  There  was  a  hint  of  an  input  device  by 
latency  interaction,  however,  F(l,  50)  =  3.12,/?  =  .08,  r)p  =  .06.  Mean  latency  ratings  for  the 
delay  and  non-delay  conditions  were  2.1  (SD  =  1.0)  and  2.5  (SD  =  1.3)  respectively,  if 
participants  used  the  game  controller;  but  were  3.4  (SD  =  1 .6)  and  2.6  (SD  =  0.9)  respectively,  if 
participants  used  the  mouse. 

Mission  1 

Mission  1  was  similar  to  Practice  Exercise  5,  except  there  were  no  waypoints  on  the 
satellite  map  and  no  time  limit.  Although  participants  were  instructed  to  avoid  collisions,  4 
participants  had  a  collision  on  their  first  attempt.  All  but  one  succeeded  on  their  second  attempt, 
and  that  person  succeeded  on  their  third  attempt.  The  ANCOVA  was  performed  on  collision-free 
completion  times.  These  were  significantly  affected  by  input  device,  F(1 ,  50)  =  7.49,/?  =  .01 ., 
r|p2  =  .13.  Participants  using  the  game  controller  completed  the  mission  significantly  faster  (mean 
203.3  s;  SD  =  16.2)  than  participants  using  the  mouse  (mean  213.8  s;  SD  =  1 1.9).  Latency  failed 
to  have  a  significant  effect  on  completion  time,  F(l,50)  =  1.11,/?  =  .30.  Mean  completion  time 
for  the  no-latency  condition  was  206.6  s  (SD  =  15.5),  and  the  mean  completion  time  for  the 
latency  condition  was  210.5  s  (SD  =  14.7). 

The  mean  subjective  workload  for  Mission  1  (across  conditions)  was  38.4  (SD  =  19.8). 
The  ANCOVA  was  performed  on  workload  scores.  VGE  was  found  to  be  a  significant  covariate 
of  workload  /7’(  1 ,50)  =  13.22,/?  =  .00,  r|p2  =  .21.  Higher  VGE  was  associated  with  lower  reported 
workload  scores.  Input  device  and  latency  each  failed  to  have  a  significant  effect  on  workload 
scores. 


Latency  ratings  were  skewed  to  the  left  and  therefore  analyzed  non-parametrically. 
Latency  ratings  were  significantly  affected  by  latency  condition,  U  =  268.5,/?  =  .035.  Mean 
latency  rating  for  the  delayed  group  was  2.7  (SD  =  1.3),  whereas  mean  latency  rating  for  the  non- 
delayed  group  was  2.0  (SD  =  1.0). 

Mission  2 

Mission  2  was  a  repeat  of  practice  exercise  6,  except  there  were  no  waypoints  on  the 
satellite  map  and  no  time  limit.  Although  participants  were  instructed  to  avoid  collisions,  1 5 
participants  had  a  collision  on  their  first  attempt.  Participants  with  collisions  were  very  evenly 
spread  among  the  four  conditions  (3  from  the  mouse-delay  condition,  and  4  each  in  the 
remaining  three  conditions).  For  the  15  people  who  restarted  the  mission,  10  were  successful  on 
their  second  attempt.  Four  were  successful  on  the  third  attempt,  and  the  final  person  succeeded 
on  their  fourth  attempt. 

The  completion  times  for  collision-free  Mission  2  violated  the  ANCOVA  assumption  of 
homogeneity  of  slopes.  The  results  of  a  Mann-Whitney  U  test  indicated  that  participants  using 
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the  game  controller  completed  these  missions  faster  (mean  238.3s;  SD  =  30.8)  than  participants 
using  the  mouse  (254.3  s;  SD  =  21.6),  U  =  236,  p  =  .01 .  There  failed  to  be  any  significant  effect 
of  latency,  U  =  384.5,  p  =  .90.  Mean  completion  time  for  the  no-latency  condition  was  245.8  s 
(SD  =  30.1),  and  the  mean  for  the  latency  condition  was  246.8  s  (SD  =  25.4). 

The  mean  subjective  workload  for  Mission  2  (across  conditions)  was  43.8  (SD  =  20.8). 
The  ANCOVA  was  performed  on  workload  scores.  VGE  was  found  to  be  a  significant  covariate 
of  subjective  workload  for  Mission  2,  /r(  1,50)  =  4.54,  p  =  .04,  t|p2  =  .08.  Higher  VGE  was 
associated  with  lower  scores.  In  addition,  spatial  ability  was  found  to  be  a  significant  covariate  of 
workload,  /r(  1 ,50)  =  6.0 9,p  =  .02,  r|p  =.11.  Higher  spatial  ability  scores  were  associated  with 
lower  workload  scores.  Input  device  and  latency  failed  to  have  significant  effects  on  workload 
scores.  Mean  latency  rating  across  conditions  for  Mission  2  was  2.3  (SD  =  1.2),  and  failed  to  be 
affected  significantly  by  input  device  or  latency. 

Mission  3 

All  participants  were  given  seven  minutes  to  complete  as  much  of  the  mission  as 
possible.  The  performance  dependent  variable  was  the  number  of  targets  photographed.  In 
scoring  performance,  one  point  was  awarded  for  each  entity  that  was  photographed  with  both 
cameras  (lA  point  for  each  camera  view).  The  maximum  possible  score  for  targets  photographed 
was  14.  The  number  of  targets  photographed  was  significantly  correlated  with  VGE  (Spearman  r 
=  .27).  The  ANCOVA  was  performed  on  the  number  of  targets  photographed.  Input  device  had  a 
marginally  significant  effect  on  targets  photographed,  /r(  1,50)  =  3.08,/?  =  .09,  r|p2  =  .06,  with 
participants  using  the  game  controller  photographing  more  targets  (mean  7.0;  SD  =  2.4)  than 
participants  using  the  mouse  (mean  6.0;  SD  =  2.2).  Latency  had  no  significant  effect  on  targets 
photographed,  F(l,50)  =  2.36, p  =  .13.  The  means  were  6.0  (SD  =  2.2)  and  6.9  (SD  =  2.4)  for  the 
latency  and  no-latency  conditions,  respectively. 

After  landing,  participants  were  asked  to  recall  the  type  and  location  of  targets  observed 
during  the  mission.  To  do  this,  they  marked  target  locations  on  a  scale  map  of  the  terrain.  One 
point  was  awarded  for  each  target  the  participant  correctly  identified  within  a  2”  diameter  circle. 
Two  points  were  assigned  for  each  target  correctly  identified  within  a  1 .25”  diameter  circle.  A 
point  was  awarded  for  each  vehicle  correctly  identified.  On  this  map,  all  of  the  targets  faced 
north.  Another  point  overall  was  awarded  for  identifying  this  orientation.  The  maximum  possible 
score  for  this  task  was  43.  The  mean  score  on  this  map  test  was  22.6  (SD  =  6.8)  in  the  game 
controller  condition  and  18.5  (SD  =  6.8)  in  the  mouse  condition.  The  post-test  scores  for  Mission 
3  violated  the  ANCOVA  assumption  of  homogeneity  of  slopes;  according  to  a  Mann- Whitney  U 
test  this  difference  was  significant  for  input  device,  U  =  263,  p  =  .03;  but  a  test  for  the  effect  of 
latency  failed  to  indicate  any  impact,  U  =  325.5,  p  =  .28. 

If  participants  collided  with  an  object  during  this  mission,  they  were  not  restarted; 
instead,  the  number  of  total  collisions  was  recorded.  Twenty-eight  of  the  56  participants  collided 
at  least  once  during  the  mission,  and  the  mean  number  of  collisions  was  2.5  (SD  =  3.8).  The  total 
number  of  collisions  was  significantly  correlated  with  the  number  of  targets  photographed 
(Spearman  r  =  -.27)  and  spatial  ability  (Spearman  r  =  -.29),  although  there  were  no  significant 
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effects  of  input  device  or  delay.  Higher  spatial  ability  yielded  fewer  collisions,  and  fewer 
collisions  were  associated  with  a  greater  number  of  identified  targets. 

The  mean  subjective  workload  across  conditions  for  this  reconnaissance  mission  was 
60.3  (SD  =  19.8).  The  ANCOVA  was  performed  on  workload  scores.  While  spatial  ability  was 
not  a  significant  covariate,  VGE  was  a  significant  covariate  of  workload  F(l,50)  =  4.12,/?  =.05, 
rip2  =  .08.  Higher  VGE  was  associated  with  lower  workload  scores.  Neither  input  device  not 
latency  produced  significant  effects  on  workload. 

The  mean  latency  rating  across  conditions  for  this  reconnaissance  mission  was  3.3  (SD  = 
1 .7).  The  latency  scores  for  Mission  3  violated  the  ANCOVA  assumption  of  normality,  and 
nonparametric  tests  failed  to  indicate  an  effect  of  input  device  or  latency. 

Summary  of  replicated  missions. 

The  results  of  input  device  in  this  experiment  were  almost  identical  to  those  obtained  in 
our  previous  research.  Table  6  shows  that  impacts  of  input  device  were  the  same  for  each 
mission,  except  for  Mission  3.  In  the  current  experiment,  an  advantage  for  the  game  controller 
not  seen  in  the  previous  experiment  was  detected.  In  the  present  experiment,  participants  were 
able  to  better  recall  Mission  3  targets  and  place  them  accurately  on  a  paper  map  if  they  had 
conducted  the  mission  with  the  game  controller  as  opposed  to  the  mouse.  This  latter  result  seems 
relevant  to  the  issue  of  whether  better  spatial  learning  might  underlie  the  advantages  of  using  the 
game  controller,  because  better  spatial  learning  should  facilitate  performance  in  these  recall  task. 
The  results  for  Mission  4  and  5,  in  a  novel  environment,  address  this  question. 


Table  6.  Comparison  of  Significant  Effects  of  Input  Device  (+)  for  Identical  Missions  in  Current 
and  Previous  Research 


Previous  Experiment 

Current  Experiment 

Practice 

Workload 

(not  reported) 

+ 

Mission  1 

Completion  time 

+ 

+ 

Collisions 

0 

0 

Workload 

0 

0 

Mission  2 

Completion  time 

+ 

+ 

Collisions 

0 

0 

Workload 

0 

0 

Misison  3 

Photo-score 

0 

0 

Map-score 

0 

+ 

Collisions 

0 

0 

Workload 

0 

0 

Mission  4  (Transfer  of  Flight  Skills  to  a  Novel  Environment) 

During  Mission  4,  only  four  participants  had  a  collision,  and  out  of  these,  only  one 
participant  collided  more  than  once.  The  ANCOVA  performed  on  completion  times  indicated 
that  these  times  were  significantly  affected  by  input  device,  F(l,  50)  =  14.51,/?  =  .00,  r|p  =  .23. 
Completion  took  longer  in  the  mouse  condition  (mean  259.5  s;  SD  =  19.8)  than  in  the  game 
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controller  condition  (mean  232.9  s;  SD  =  30.0).  Latency  had  a  marginally  significant  effect  on 
completion  times,  F(1 ,50)  =  2.97,  p  =  .09,  r|p  =  .06;  participants  without  latency  completed  the 
mission  faster  (mean  240.4  s;  SD  =  26.4)  than  participants  with  latency  (mean  252.0  s;  SD  = 
29.9). 


The  mean  subjective  workload  for  Mission  4  (across  conditions)  was  43.2  (SD  =  20.9). 
The  mean  latency  rating  score  was  2.4  (SD  =  1 .2).  Both  measures  violated  the  ANCOVA 
assumption  of  homogeneity  of  slopes,  and  results  of  nonparametric  tests  failed  to  indicate  any 
significant  effects. 

The  data  collected  from  this  transfer  skills  mission  demonstrate  that  the  temporal 
advantage  of  the  game  controller  over  the  mouse  is  not  specific  to  the  training  environment. 
Mission  completion  time  differences  were  still  evident  even  when  participants  were  placed  in  a 
novel  environment.  These  results  suggest  that  differences  in  spatial  learning  cannot  account  for 
the  performance  differences. 

Mission  5  (Ti ransfer  Tactical  Mission  Performance  to  a  Novel  Environment) 

Our  previous  research  failed  to  find  a  difference  in  number  of  targets  photographed  or 
recalled  across  the  different  input  device  conditions,  when  mission  duration  was  fixed. 
Nevertheless,  we  decided  to  include  a  similar  tactical  mission  in  the  transfer  environment  in  this 
experiment.  It  is  possible  that  demands  of  negotiating  a  relatively  new  environment  might  reveal 
some  differences  due  to  input  device  that  were  not  detectable  in  the  more  familiar  environment. 
Mission  5  was  a  tactical  exercise  much  like  Mission  3.  Twelve  entities  (all  Soldiers)  were 
imported  into  the  terrain  database,  and  participants  were  given  seven  minutes  (although  they  did 
not  know  the  duration)  to  identify  as  many  of  these  entities  as  possible.  Participants  were 
required  to  take  a  photograph  of  each  entity’s  face  with  the  forward  camera  in  order  to  achieve 
positive  identification.  In  scoring  performance,  one  point  was  awarded  for  each  entity  that  was 
photographed.  The  maximum  possible  score  for  targets  photographed  was  12.  The  resulting  data 
violated  ANCOVA  assumptions,  and  nonparametric  tests  failed  to  reveal  any  significant  effects 
of  input  device  or  latency.  Mean  targets  photographed  was  7.9  (SD  =  1 .9).  This  measure  was 
significantly  correlated  with  VGE  (Spearman  r  =  .28). 

After  landing,  participants  were  asked  to  recall  the  location  of  targets  observed  during  the 
mission.  To  do  this,  they  marked  target  locations  on  a  scale  map  of  the  terrain.  One  point  was 
awarded  for  each  target  the  participant  correctly  identified  within  a  2”  diameter  circle.  Two 
points  were  assigned  for  each  target  correctly  identified  within  a  1.25”  diameter  circle.  If  a 
participant  correctly  identified  an  entity’s  vertical  location  (ground,  2nd  floor,  or  tower),  he 
received  an  additional  point.  Further,  one  point  was  assigned  for  each  entity  that  was  correctly 
marked  as  either  armed  or  unarmed.  The  maximum  possible  score  for  this  task  was  48.  The  mean 
number  of  targets  correctly  recalled  on  the  map  was  1 8.3  (SD  =  1 2.3).  The  scores  on  this  test 
violated  the  ANCOVA  assumption  of  normality.  Nonparametric  tests  failed  to  reveal  any 
significant  effects  of  input  device  or  latency. 

Sixteen  of  the  56  participants  had  one  or  more  collisions  during  Mission  5.  There  failed 
to  be  any  relation  between  collisions  and  condition.  The  mean  subjective  workload  for  Mission  5 
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(across  conditions)  was  51 .4  ( SD  =  1 8.9).  The  workload  ratings  violated  the  ANCOVA 
assumption  of  homogeneity  of  slopes,  and  nonparametric  testing  failed  reveal  any  significant 
effects. 


Latency  ratings  were  higher  in  the  delay  condition  (mean  2.9;  SD  =  1 .6)  than  the  no-delay 
condition  (mean  2.1 ;  SD  =  1.1),  and  this  difference  was  significant  according  to  the  ANCOVA, 
F(  1 ,  50)  =  4.57,  p  =  .04,  r|p  =  .08.  There  failed  to  be  an  effect  of  input  device  on  latency  rating. 

The  data  collected  from  Mission  5  show  no  significant  effects  of  input  device  on 
performance;  however,  such  results  were  expected  based  on  our  previous  research.  One  possible 
interpretation  is  that  input  device  is  not  that  influential  when  free  flight  is  permitted,  and  the  pilot 
does  not  have  to  follow  a  prescribed  course.  Another  possibility  is  that  our  measures  in  the 
tactical  missions  (i.e.,  number  of  targets  photographed  and/or  recalled  in  a  fixed  time)  are 
relatively  insensitive,  compared  to  the  temporal  measures  used  in  the  flight  skill  missions. 


Usability  Questionnaire 

The  Usability  Questionnaire  was  administered  after  the  final  mission.  Participants  were 
asked  to  rate  various  aspects  of  the  MAV  simulation,  using  a  Likert  scale,  where  1  was  the  most 
favorable  rating  and  10  was  the  least  favorable  rating.  Averaged  over  all  usability  questions,  the 
mean  usability  score  was  3.0  (SD  =  0.8).  A  repeated  measures  ANCOVA  on  Usability  Questions 
revealed  an  interaction  of  question  and  input  device,  F(31,  1550)  =  9.93,  p  =  .00,  r|p2  =  .17,  and 
also,  question  and  VGE,  F(31,  1550)  =  1.51,/?  =  .04,  r|p2  =  .03.  Therefore,  we  performed 
separate  analyses  of  individual  questions.  Table  7  lists  the  questions  for  which  the  participants  in 
the  game  controller  condition  rated  the  item  significantly  more  favorably  (lower  score)  than 
participants  in  the  mouse  condition,  whereas  Table  8  lists  the  questions  with  opposite  results. 


Table  7.  Mean  Usability  Scores  for  Questions  for  which  Participants  using  the  Game  Controller 


Usability  Question 

Mean 

Mouse 

SD 

Mouse 

Mean 

GC* 

SD 

GC* 

F(  1,50) 

6)  The  system  I  worked  with 
was... 

1  flexible— . rigid  10 

4.4 

2.2 

3.2 

1.7 

5.77,  p  =  .02 

.10 

21)  The  system  provided 
adequate  feedback  when  I  issued 
a  command  to  the  MAV. 

1  always - never  1 0 

3.4 

2.1 

2.1 

1.3 

8.50,  />  =  .  01 

.15 

29)  The  responsiveness  of  the 
system  to  my  input  was. . . 

1  fast  enough-too  slow  10 

4.6 

2.7 

3.1 

1.5 

7.35,  p  =  .  01 

.13 
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Table  8.  Mean  Usability  Scores  for  Questions  for  which  Participants  Using  the  Mouse  Gave 
Significantly  more  Favorable  Ratings  (Lower  Scores)  than  Participants  Using  the  Game 
Controller,  along  with  Corresponding  Statistics 


Usability  Question 

Mean 

Mouse 

SD 

Mouse 

Mean 

GC* 

SD 

GC* 

FT  1.50) 

V 

7)  The  functions  of  the  on-screen 
manual  control  buttons  (on  the 
control  pad)  were. . . 

1  clear- . confusing  10 

1.4 

0.9 

2.2 

1.9 

4.92,  p  =  .03 

.09 

12)  As  I  progressed  through  the 
missions  using  the  [joystick/mouse], 
my  hands  and/or  wrists  became 
fatigued. 

1  never- . always  1 0 

1.7 

1.3 

8.4 

2.1 

191.40, p  =  .00 

.79 

*GC  =  game  controller 


In  our  previous  research  (2008),  we  found  that  nine  questions  were  rated  more  favorably 
from  the  game  controller  condition,  including  questions  pertaining  to  ratings  of  the  system  itself, 
feedback,  MAV  control,  and  awareness  of  exercise  objectives.  In  the  current  research, 
participants  in  the  game  controller  condition  rated  three  questions  more  favorably  than 
participants  in  the  mouse  condition.  The  findings  for  only  one  question  were  identical  over  both 
studies,  and  this  question  related  to  feedback  that  was  given  after  a  command  had  been  issued. 
Previous  research  also  showed  that  the  mouse  condition  rated  one  question  (which  was  related  to 
system  speed)  significantly  more  favorably  than  the  game  controller  condition.  This  was  not 
found  in  our  current  research.  Instead  we  found  that  a  question  relating  to  physical  fatigue  was 
rate  significantly  more  favorably  by  the  mouse  condition.  This  is  surprising  because  workload 
scores  throughout  the  experiment  did  not  reflect  this  difference  between  input  devices. 

VGE  was  found  to  be  a  significant  covariate  of  one  item.  When  asked  to  rate  how  clear  or 
confusing  the  display  was,  participants  with  higher  VGE  gave  more  favorable  ratings,  i7(l,50)  = 
4.94 ,p  =  .03,  rjp  =  .09,  Pearson  r  =  .30.  This  was  not  found  in  our  previous  research  (Durlach, 
Neumann,  &  Billings,  2008).  In  addition,  there  was  an  overall  lack  of  effect  of  latency  for  each 
question. 


Failures 

Out  of  66  participants  who  began  the  experiment  10  females  failed  to  meet  practice 
exercise  criteria.  Examining  demographic  and  spatial  ability  data,  the  only  significant  difference 
between  this  group  and  those  who  completed  the  experiment  was  their  self-reported  video  game 
skill.  For  the  question  “How  would  you  rate  your  video  game  skills — (1)  novice/beginner,  (2) 
intermediate,  or  (3)  expert?”,  the  group  who  completed  the  experiment  rated  themselves  with 
significantly  higher  video  game  skills  (mean  1 .7;  SD  =  0.7)  than  the  group  who  did  not  meet  the 
practice  criteria  (mean  1.2;  SD  =  0.4),  U  =  165 ,p  =  .04. 
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Conclusions 


In  our  experiment,  we  intended  to  replicate  and  extend  our  previous  research,  which 
suggested  that  operator  performance  was  quicker  and  more  efficient  when  using  a  game 
controller  rather  than  a  mouse  (Durlach,  Neumann,  &  Billings,  2008).  Faster  mission  completion 
is  an  important  concern  for  MAV  operators  because  smaller  UAVs  have  restricted  air  time  due  to 
limited  battery  and/or  fuel  capacity.  The  findings  of  the  present  research  replicated  the  findings 
of  our  previous  research.  Performance  advantages  of  the  game  controller  over  the  mouse  were 
manifest  in  quicker  completion  times  in  both  the  training  environment  and  the  novel 
environment.  The  fact  that  the  performance  benefits  of  the  game  controller  carried  over  to  the 
novel  environment  suggests  that  the  difference  due  to  input  device  cannot  be  attributed  to 
differential  spatial  learning. 

We  previously  interpreted  the  benefits  to  of  the  game  controller  to  the  ability  it  affords  to 
continuously  control  maneuver  in  multiple  dimensions.  Conversely  the  mouse  affords  discrete 
control  in  one  direction  at  a  time.  There  are  other  differences  between  the  game  controller  and 
the  mouse  that  could  contribute  to  the  performance  difference,  however.  One  important 
difference  is  the  division  of  visual  attention.  The  mouse  requires  the  operator  to  constantly 
reorient  visual  attention,  between  the  sensor  image  and  the  interface  control  display  (arrows).  In 
contrast,  for  the  game  controller,  once  its  functions  are  mastered,  the  operator  can  focus  attention 
entirely  on  the  sensor  imagery.  As  there  are  no  visual  icons  that  must  be  “clicked  on,”  there  is  no 
need  to  attend  to  the  interface  control  display  (although  it  did  provide  feedback).  It  is  this 
difference  in  attention  to  the  sensor  image  which  we  hypothesized  might  account  for  a  difference 
in  spatial  learning.  Although  the  present  data  suggest  that  a  difference  in  spatial  learning  did  not 
account  for  the  performance  difference,  it  still  may  be  that  the  opportunity  to  focus  visual 
attention  more  completely  on  the  sensor  image  accounted  for  the  benefits  of  the  game  controller. 
Such  focused  attention  should  lower  cognitive  demands  and  opportunities  for  change  blindness 
(Durlach,  2004;  Rubinstein,  Meyer,  &  Evans,  2001 ;  Wickens  &  Liu,  1988).  Perhaps  a  better- 
designed  input  control  display  (e.g.,  integrated  display  with  visual  overlay)  would  reduce 
operator  performance  differences.  However,  even  with  a  change  in  the  display,  the  game 
controller  allows  more  focused  attention  because  there  is  little  reason  to  deviate  from  the  sensory 
imagery.  With  a  visual  overlay,  a  person  using  the  mouse  would  still  need  to  refocus  to  some 
extent  on  the  control  he/she  is  clicking. 

In  our  current  research,  the  benefits  of  the  game  controller  were  primarily  restricted  to  the 
flight  skill  missions.  One  exception  was  that  for  Mission  3,  those  using  the  game  controller  were 
able  to  more  accurately  report  target  characteristics  and  locations  post-mission.  It  is  possible  that 
the  flight  skill  missions,  with  their  heavy  emphasis  on  speed,  maneuver,  and  obstacle  avoidance 
may  have  unique  sensitivity  to  reveal  performance  differences  due  to  input  device.  On  the  other 
hand,  it  is  possible  that  the  difference  in  sensitivity  lies  not  in  the  mission  requirements,  but  in 
the  dependent  variables  used  to  measure  performance.  The  temporal  measures  were  taken  on  an 
interval  scale  without  restricted  range.  The  measures  used  to  count  targets  detected  and  collisions 
were  taken  on  an  ordinal  scale.  The  collisions  measure,  in  particular  was  poorly  suited  to 
statistical  analysis,  consisting  of  mostly  0’s  and  1  ’s.  Perhaps  tactical  missions  that  are  conducted 
to  completion  (all  targets  detected)  and  timed  would  reveal  performance  benefits  of  the  game 
controller  over  the  mouse  as  well. 
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With  regard  to  latency  effects,  no  significant  effects  of  delay  on  performance  were  found. 
Neither  did  latency  seem  to  affect  subjective  workload.  Significant  subjective  reports  of  latency 
were  found,  for  Missions  1  and  5.  The  500  ms  latency  condition  reported  noticing  latency  to  a 
greater  extent  than  the  no-latency  condition.  It  seems  therefore,  that  the  500  ms  latency  did  not 
have  much  of  a  detectable  detrimental  effect,  and  of  most  relevance  for  the  present  research,  did 
not  undermine  the  performance  benefits  of  the  game  controller  over  the  mouse. 

Research  should  consolidate  and  address  all  of  the  necessary  control  mechanisms  for 
MAV  operation.  Many  control  mechanisms  for  vehicle  operation  do  not  function  in  isolation  of 
other  control  devices.  For  instance,  in  our  research  we  did  not  look  at  keyboard  usage  in 
conjunction  with  MAV  operation,  yet  realistically  this  may  be  a  mode  of  communication 
between  commander  and  MAV  operator  in  the  field.  As  such,  it  would  be  advantageous  to 
design  a  system  that  incorporates  all  control  mechanisms  on  the  same  operator  control  unit  so 
that  switching  between  mechanisms  is  minimized. 

Future  research  should  also  continue  to  address  the  impact  of  input  device  as  well  as  the 
different  levels  of  system  latency  on  performance  in  the  operation  of  unmanned  systems.  As  the 
military  increases  its  use  of  these  systems,  human-system  interaction  will  be  become  an 
increasingly  important  area  of  concern.  Our  experiments  were  conducted  in  a  lab  with  the 
participant  sitting  at  a  desk.  We  did  not  manipulate  any  environmental  conditions  such  as 
weather,  which  can  affect  system  performance.  Therefore,  the  results  we  have  presented  favoring 
continuous  over  discrete  input  devices  must  be  tested  for  resiliency  in  more  realistic  usage 
environments. 
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Appendix  A 


Acronyms 


ANCOVA 

Analysis  of  Covariance 

ARI 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 

DARPA 

Defense  Advanced  Research  Projects  Agency 

FCS 

Future  Combat  Systems 

ICD 

Input  Control  Display 

IED 

Improvised  Explosive  Devices 

GPS 

Global  Positioning  System 

MAV 

Micro-Aerial  Vehicle 

NAV 

Nano-Aerial  Vehicle 

ocu 

Operator  Control  Unit 

ONESAF 

One  Semi-Automated  Forces 

RSTA 

Reconnaissance,  Surveillance,  and  Target  Acquisition 

SD 

Standard  Deviations 

TLX 

Task  Load  Index 

UAS 

Unmanned  Aviation  Systems 

UAV 

Unmanned  Aerial  Vehicles 

VGE 

Video  Game  Experience 

A-l 


Appendix  B 

Illustration  of  Operator  Control  Unit  Display  Configuration 


Task  bar  icons 


Heading  tape 


Mission  timer 


video  sensor 
imagery 


Control  pad 
_ 


Overhead  map 


Figure  B-l .  OCU,  single  view  camera  layout  with  discrete  input  control. 
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Appendix  C 

Participant  Training  Guide 

A  training  manual  was  produced  for  the  game  controller,  and  a  different  training  manual  was 
created  for  the  mouse.  To  create  these,  an  original  “baseline”  guide  was  edited  to  take  into 
account  the  required  variations  for  each  condition. 

The  following  gives  the  training  manual  for  participants  in  the  game  controller  conditions. 
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Army  Research  Institute 
Intuitive  Means  of  Robotic  Control 
Testbed 


Participant’s  Guide 

GC/Joystick 


INSTITUTEfor 
SIMULATION 
Q TRAINING 


MAV  Study  -  1ST  1 
Spring/Summer  2007 
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Introduction  to  the  Micro-Unmanned  Aerial  Vehicle  Study 


The  U.S.  Army  is  undergoing  a  major  transformation.  One  element  of  the  transformation 
is  the  introduction  of  a  new  class  of  military  platforms  known  as  unmanned  air  and 
ground  vehicles  ( called  UAVs  and  UGVs).  A  major  benefit  of  these  unmanned  vehicles 
is  that  they  can  perform  reconnaissance  missions  and  survey  areas  contaminated  with 
radiological,  chemical,  or  biological  agents  without  risk  to  human  life.  They  can  also 
survey  the  battlefield  and  provide  real-time  video  feedback. 

We  are  investigating  the  design  of  operator  control  systems  for  micro-unmanned  aerial 
vehicles  that  can  perform  these  kinds  of  reconnaissance  missions.  In  addition,  we  are 
investigating  operator  training  requirements.  In  this  experiment,  you  will  be  trained  on 
how  to  fly  a  simulated  micro-UAV  (MAV),  and  then  you  will  complete  a  set  of  missions 
that  will  test  your  ability  to  maneuver  the  MAV  and  locate  various  targets.  After  each 
mission,  you  will  be  given  a  short  questionnaire  that  asks  you  to  rate  certain  aspects  of 
the  task  you  performed. 


We  have  allotted  approximately  3  hours  for  this  experiment.  No  previous  flight 
experience  is  necessary  to  participate  in  this  study. 


Confidentiality 

Your  identity  will  be  kept  confidential  to  the  extent  provided  by  law.  Your  information  will 
be  assigned  a  code  number.  The  list  connecting  your  name  to  this  number  will  be  kept 
in  an  electronic  file.  When  the  study  is  completed  and  the  data  have  been  analyzed,  the 
list  will  be  destroyed.  Your  name  will  not  be  used  in  any  report. 

If  you  are  prepared  to  participate  in  this  experiment,  please  read  and  sign  the  Consent 
Form  and  Voluntary  Agreement.  Please  also  indicate  on  that  form  your  preferred 
method  of  compensation.  We  offer  cash  payment  or  course  credit.  Also,  please  feel  free 
to  ask  the  experimenter  any  questions.  Keep  in  mind  that  you  do  have  the  right  to 
withdraw  from  this  experiment  at  any  time,  for  whatever  reason. 

When  you  have  finished  reading  and  signing  the  voluntary  consent  form,  please  return  it 
to  the  experimenter. 
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Overall  Description  of  the  MAV  Simulation 


You  will  be  working  with  a  simulation  for  flying  a  micro-unmanned  aerial  vehicle.  The 
micro-UAV  itself  will  be  referred  to  from  now  on  as  “the  MAV.”  This  is  not  a  fixed-wing 
aircraft  like  most  airplanes  are;  rather,  it  is  a  small  rotary  craft  with  an  internal  fan  and 
duct  design  (see  prototype  photos  below).  An  operator  controls  the  MAV  using  a  laptop 
computer  equipped  with  an  input  device  such  as  a  mouse  or  joystick  controller.  This 
interface  is  referred  to  as  the  OCU  -  Operator  Control  Unit.  This  is  a  dismounted  control 
unit,  as  it  is  envisioned  that  a  dismounted  Soldier  (on  foot  rather  than  in  a  vehicle)  will 
be  controlling  the  MAV. 


MAV  prototypes  from  Honeywell  and  Allied  Aerospace  . 


Introduction  to  the  OCU  and  MAV  Camera  System 

The  MAV  is  equipped  with  a  dual  camera  system.  When  the  vehicle  flies  through  the 
simulated  environment  you  will  be  able  to  view  video  images  sent  back  to  the  OCU.  You 
will  be  instructed  on  how  to  operate  the  cameras  as  well  as  how  to  use  the  OCU 
interface  and  controllers  to  pilot  the  MAV.  You  will  also  have  an  opportunity  to  practice 
some  manual  flight/piloting  techniques  before  beginning  the  actual  experiment.  After 
basic  instruction,  a  training  session  will  take  place;  then  you  will  move  on  to  the 
assigned  pilot  mission  tasks  where  performance  data  will  be  recorded.  Be  sure  that  you 
understand  the  objective  of  each  mission  before  starting  a  trial.  The  experimenter  is 
available  to  answer  your  questions  before  you  begin  each  task,  so  please  ask  for  help  if 
you  are  unsure  of  any  requirements.  Unless  instructed  otherwise,  it  is  important  that 
you  complete  each  task  as  quickly  and  efficiently  as  possible. 

At  the  end  of  the  training  session  and  at  the  end  of  each  mission  you  will  complete  a 
short  computer-based  questionnaire.  In  the  first  section  you  will  rate  different  aspects  of 
the  task  you  performed;  then  you  will  be  asked  to  choose  between  a  pair  of  items  that 
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relate  to  your  performance.  You  must  choose  one  and  only  one  item  for  each  pair.  Then 
you  will  proceed  to  the  next  mission. 

Please  let  the  experimenter  know  when  you  are  finished  reading  these  sections. 

Training  Session 


The  goal  of  training  is  to  familiarize  you  with  the  flight  characteristics  of  the  MAV  and  to 
give  you  an  opportunity  to  practice  piloting  the  MAV  in  manual  mode.  We  will  begin  by 
reviewing  the  features  of  the  OCU  and  then  proceed  to  a  series  of  practice  exercises. 
The  experimenter  will  facilitate  this  training  session  and  provide  instruction  on  how  to 
complete  the  assigned  tasks. 

OCU  Layout  and  On-Screen  Controls 

Below  you  will  see  a  sample  layout  of  the  OCU.  On  the  left  side  of  the  screen  is  the 
video  sensor  imagery  (camera  views),  and  in  the  upper-right  side  you  will  find  an 
overhead  map  view  of  the  terrain  database— this  is  the  satellite  view  of  the  MAV.  Just 
below  the  map  view  there  is  a  control  pad  that  is  used  for  issuing  flight  commands  to  the 
MAV.  The  control  pad  will  be  examined  in  greater  detail  throughout  training.  There  is  an 
altimeter  along  the  left  border  area  of  the  OCU  display  as  well  as  a  task  bar  along  the 
top  that  contains  various  icons.  You  will  be  instructed  on  how  to  use  all  relevant  gauges 
and  icons  during  training. 


OCU  with  single  view  overlapping  cameras  with  discrete  input  control  pad 
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The  experimenter  will  handle  tasks  such  as  loading  mission  files  and  scenarios,  so  you 
only  need  to  focus  on  learning  how  to  operate  the  MAV  itself.  Before  beginning  the  flight 
training  exercises,  we  will  learn  about  the  function  of  the  OCU  in  more  detail. 

Understanding  the  Task  Bar  Icons 

The  upper  task  bar  (shown  below)  includes  the  take-off  (and  landing),  mission 
mode,  and  camera  control  buttons.  There  is  also  a  mission  timer  located  on  the  far 
right  of  this  task  bar.  For  this  study,  you  will  need  to  know  how  to  use  the  take-off  icon 
and  the  camera  control  icons. 


Take-off  &  Landing  icon 


OifD;  You  can  take  off  and  land  by  pressing  button  (10)  on  the  joystick  controller  to 
activate  the  task  bar  icon  for  take-off  and  landing.  ALSO,  button  (10)  acts  as  the  “OK" 
button  to  begin  the  missions. 

Once  the  take-off  button  is  pressed,  the  MAV  will  automatically  climb  to  an  altitude  of 
approximately  60  feet  above  the  ground  level.  You  will  see  a  red  stop  sign  icon 
illuminate  when  you  have  reached  take-off  altitude.  Pressing  the  take-off  button  again 
will  execute  the  land  command.  You  may  land  the  MAV  now. 

Activating  Camera  Views  and  Taking  Snapshot  Photos 


rot®® 


These  camera  buttons  allow  the  operator  to  switch  between  the  two  available  camera 
images.  This  study  uses  a  two  camera  setup.  On  the  OCU,  camera  image  #1  is  the 
view  from  the  MAV’s  forward  camera,  and  camera  image  #2  displays  the  view  from  the 
downward  camera.  In  this  experiment,  one  camera  view  will  be  displayed  at  a  time  so 
there  will  be  a  need  for  the  operator  to  switch  views  to  activate  a  camera  for  taking 
snapshots. 

One  of  the  unique  features  of  the  OCU  is  the  ability  to  take  snapshots  with  either 
camera.  Before  taking  a  snapshot,  you  will  need  to  activate  the  camera  that  you  want  to 
take  the  picture.  The  active  camera  image  will  always  have  a  ( )  overlay  in  the  center  of 
the  image.  The  corresponding  icon  on  the  task  bar  will  also  illuminate.  To  change  the 
active  camera  view,  you  must  use  the  joystick  to  activate  the  group  of  icons  on  the  task 
bar  and  select  the  camera  you  want. 
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To  activate  a  camera  view:  This  is  done  by  pressing  and  holding  joystick  button  (2) 
while  you  scroll  through  the  available  camera  views  with  the  directional  pad.  The 
experimenter  will  demonstrate  this  feature  now. 

If  not  already  airborne,  try  taking  off  and  switching  the  camera  view.  Snapshots  of 
targets  can  now  be  taken  using  button  (9)  on  the  game  controller/joystick. 


Main  Window  Components 

Altimeter  -  (See  vertical  bar  on  the  left  side  of  this  page) 

The  ruler-like  markings  on  the  altimeter  displays  the  altitude  of  the  MAV  in  feet  above 
sea  level.  Red  tabs  may  be  visible  on  the  upper  and  lower  regions  of  the  display  if  the 
experimenter  has  chosen  to  activate  the  altitude  alarm  system.  The  red  areas  simply 
mark  the  altitudes  that  will  activate  the  alarm  if  the  MAV  passes  into  this  "red  zone." 


<  The  white  triangle  cut-out  (left)  points  to  the  current  altitude  of  the  MAV.  In  this  case, 
the  MAV  is  approximately  82  feet  above  sea  level. 


The  light  brown  column  at  the  lower  end  of  the  bar  marks  the  altitude  of  the  nearest 
surface  below  the  MAV  (this  is  the  current  ground  level). 


I  Note!~  In  the  current  example,  the  MAV  is  approximately  82  feet  above  sea  level,  but 
1  the  ground  level  is  approximately  22  feet.  This  means  the  MAV  is  only  60  feet  above 
!  ground!  . 


Discrete  Manual  Input  Control  Pad 


This  control  pad  lets  you  control  the  position  of  the  MAV  manually.  For  this  display, 
nine  buttons  are  used  as  the  interface  to  the  MAV.  The  four  arrow  icons  move  the  MAV 
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forwards,  backwards,  (strafe)  left,  and  (strafe)  right.  The  curved  arrow  icons  in  the  upper 
corners  rotate  the  MAV  left  and  right.  The  lower  corner  icons  move  the  MAV  up  and 
down.  The  middle  X  icon  stops  the  MAV.  Your  airspeed  is  also  shown  here  in  knots, 
and  the  max  speed  is  6  knots.  You  will  issue  these  commands  to  the  MAV  by  using  the 
joystick/game  controller.  Joystick  training  is  next. 

Note  on  Input  Controller  Feedback 

Because  you  are  using  the  joystick  to  control  the  MAV,  the  control  display  will  activate 
when  an  input  is  received.  The  display  provides  feedback  in  this  way  to  the  operator 
that  a  command  has  been  issued  and  is  being  executed.  Arrow  icons  on  the  discrete 
control  pad  will  illuminate  when  that  command  has  been  entered  (e.g.,  when  you  push 
forward  on  the  joystick,  the  forward  arrow  will  illuminate).  The  brighter  the  icon  gets,  the 
faster  you  are  traveling. 

Joystick  /  Game  Controller 

The  joystick/game  controller  is  shown  below,  and  by  now  you  have  at  least  performed 
some  basic  tasks  with  this  device.  We  will  now  go  over  how  to  use  the  joystick  for  all  of 
the  tasks  required  during  this  experiment. 


Controlling  MAV  movement  with  the  Joystick: 

The  left  thumb  stick  controls  movement  forward,  backward,  and  sideways  (or  at 
angles),  but  the  MAV  heading  never  changes  when  the  left  thumb  stick  is  used.  Pushing 
up  on  this  thumb  stick  moves  the  MAV  forward.  Pulling  down  on  the  stick  moves  the 
MAV  backwards.  Moving  the  stick  from  side  to  side  moves  the  MAV  right  or  left  without 
altering  heading. 

The  right  thumb  stick  controls  altitude  and  headinq/rotation.  Pushing  up  on  the  stick 
increases  the  MAV’s  altitude,  and  pulling  down  decreases  altitude.  Moving  the  right 
thumb  stick  from  side  to  side  rotates  the  MAV  in  place. 

It  is  possible  (and  expected)  to  use  both  sticks  simultaneously  to  rotate,  change  altitude, 
and  move  at  the  same  time.  At  this  time,  you  may  take  off  and  practice  manipulating  the 
thumb  sticks  for  approximately  1-2  minutes. 
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Heading  Tape 


Note  that  the  camera  view  window  has  a  heading  tape  located  along  the  top  edge  of 
the  frame.  This  number  indicates  the  current  heading  of  the  vehicle  (based  on  360 
degrees)  with  regard  to  that  camera  image.  Because  the  cameras  have  been  locked  in 
place  for  this  experiment,  you  can  assume  that  the  forward  camera  view  heading  is  the 
same  as  the  MAV  heading.  So  if  the  heading  tape  reads  "270"  then  you  know  the  MAV 
is  facing  due  West. 


IM  l**  X— ■■ 


0-degrees  is  the  same  as  due  North  on  a  regular  compass.  90-degrees  =  East; 
180-degrees  =  South;  270-degrees  =  W. 

Important  Note!!  ~  The  downward  camera  view  is  locked  at  90  degrees  from  horizontal 
(this  is  essentially  straight  down).  However,  when  the  MAV  is  in  motion  the  vehicle  tilts 
in  a  similar  manner  to  a  moving  helicopter.  This  tilting  will  cause  the  downward  camera 
to  point  slightly  backwards,  thus  giving  the  operator  a  heading  reading  that  is  opposite 
of  the  forward  camera  view.  i.e.  if  the  forward  camera  heading  is  0-degrees,  then 
the  downward  camera  heading  will  read  180-degrees  only  while  the  MAV  is  in 
motion. 


Practice  Time:  Now  that  you  have  learned  all  the  functions  of  the  OCU  and  the  flight 
controls,  we  will  complete  a  series  of  practice  exercises  beginning  on  the  next  page. 
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Practice  Exercises 


These  exercises  will  give  you  a  chance  to  practice  the  various  tasks  required  to 
complete  the  missions  in  this  study. 

Warm  Up 

Start  this  warm  up  session  by  executing  a  take-off  and  briefly  practicing  the  following 
maneuvers.  You  will  notice  that  inertia  comes  into  play  when  trying  to  stop  the  MAV,  so 
you  will  need  to  learn  to  estimate  things  like  stopping  distances  and  rotational  velocity 
carry-over.  You  want  to  avoid  collisions  at  all  costs!  We  will  go  over  the  warm  up 
exercises  one  by  one.  As  I  read  the  maneuvers,  please  try  to  move  the  MAV 
accordingly  with  the  joystick: 

1)  Move  the  right  thumb  stick  up  and  down  to  make  the  MAV  ascend  and  descend. 

2)  Move  the  right  thumb  stick  side  to  side  to  rotate  the  MAV. 

3)  Move  the  left  thumb  stick  up  and  down  to  make  the  MAV  fly  forward  and 
backwards. 

4)  Move  the  left  thumb  stick  side  to  side  to  move  the  MAV  laterally.  Note  that  the 
heading  only  changes  when  you  rotate  the  MAV  with  the  right  thumb  stick. 

5)  Activate  camera  2,  and  then  activate  camera  1.  Now  switch  back  to  camera  2 
and  take  a  snapshot  [joystick  button  (9)]. 

6)  Land  the  MAV. 


Next  you  will  complete  a  series  of  timed  practice  exercises.  The  experimenter  will 
observe  these  exercises  and  determine  if  you  have  met  the  time  requirement  before 
allowing  you  to  proceed  to  the  next  exercise.  All  mission  and  properties  files  needed  for 
these  exercises  will  be  loaded  by  the  experimenter. 


Practice  Exercise  1:  DON’T  BEGIN  UNTIL  I  TELL  YOU  TO  DO  SO!  Read  through  the 
following  exercise,  and  let  me  know  when  you  have  finished. 

1 )  Press  OK  (button  1 0)  to  start  the  simulation  and  timer. 

2)  Execute  the  Take-off  command. 

3)  When  the  Red  Stop  icon  illuminates,  execute  the  Land  command. 

4)  This  must  be  completed  in  30  seconds  (:30)  or  less. 
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Practice  Exercise  2:  DON’T  BEGIN  UNTIL  I  TELL  YOU  TO  DO  SO!  Read  through  the 
following  exercise,  and  let  me  know  when  you  have  finished. 

1)  Press  OK  (button  10)  to  start  the  simulation  and  timer. 

2)  Execute  the  Take-off  command. 

3)  At  or  before  the  completion  of  take-off,  activate  the  view  window  for  camera  2 
(downward  view). 

4)  Take  a  snapshot  with  camera  2. 

5)  Activate  the  view  window  for  camera  1  (forward  view). 

6)  Take  a  snapshot  with  camera  1. 

7)  Execute  the  Land  command. 

8)  This  must  be  completed  in  40  seconds  (:40)  or  less. 


Practice  Exercise  3:  DON’T  BEGIN  UNTIL  I  TELL  YOU  TO  DO  SO!  Read  through  the 
following  exercise,  and  let  me  know  when  you  have  finished. 

1)  The  upper  altitude  alarm  will  be  set  at  150  feet  and  activated. 

2)  Press  OK  (button  10)  to  start  the  simulation  and  timer. 

3)  Execute  the  Take-off  command. 

4)  Ascend  to  150  feet  and  trigger  the  alarm  (you’ll  hear  a  warning  sound). 

5)  Immediately  descend  to  50  feet  or  below  without  hitting  the  ground. 

6)  Ascend  back  up  to  100  feet  but  less  than  150  feet. 

7)  Rotate  the  MAV  360-degrees  without  dropping  below  100  feet.  It  is  required  that 
the  heading  tape  shows  the  number  “0”  after  completing  1  rotation  with  the  MAV. 
The  “0”  must  remain  in  the  forward  camera  view  window  before  landing. 

8)  Execute  the  Land  command. 

9)  This  exercise  must  be  completed  in  1  minute  35  seconds  (1 :35)  or  less. 


Practice  Exercise  4:  Rapid  Command  Execution;  DON’T  BEGIN  UNTIL  I  TELL  YOU 
TO  DO  SO!  Read  through  the  following  exercise  (both  Part  A  and  Part  B),  and  let  me 
know  when  you  have  finished. 

For  this  exercise,  you  will  follow  a  series  of  oral  commands  issued  by  the  experimenter. 
After  take-off  and  as  soon  as  the  Red  Stop  icon  illuminates,  you  will  immediately  begin 
to  hear  a  series  of  flight  commands.  Commands  will  be  given  as  fast  as  you  can 
correctly  comply.  Once  the  correct  feedback  is  observed  from  the  OCU,  the 
experimenter  will  proceed  to  the  next  command. 

Note:  It  is  not  important  that  the  MAV  travels  any  considerable  distance.  The  purpose  of 
this  exercise  is  to  allow  you  to  learn  the  mapping  of  all  buttons  and  icons  and  their 
corresponding  functions.  The  experimenter  is  looking  mainly  for  the  correct  feedback 
from  the  OCU  control  pad  located  in  the  lower  right  of  the  display. 
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Rapid  command  execution  -  Part  A 


1 )  Press  OK  (button  10)  to  start  the  simulation  and  timer. 

2)  Execute  Take-off. 

3)  The  first  series  of  commands  after  take-off  will  be:  9  commands 

4)  This  exercise  must  be  completed  in  1  minute  5  seconds  (1:05)  or  less. 


The  experimenter  will  now  reload  the  properties  file  and  reset  the  timer. 


Rapid  command  execution  -  Part  B 

1 )  Press  OK  (button  10)  to  start  the  simulation  and  timer. 

2)  Execute  Take-off. 

3)  The  first  series  of  commands  after  take-off  will  be:  14  commands 

4)  This  exercise  must  be  completed  in  1  minute  25  seconds  (1 :25)  or  less. 


The  next  two  exercises  involve  flying  the  MAV  over  longer  distances  and  following  pre¬ 
determined  mission  parameters.  These  will  be  similar  to  the  missions  you  will  complete 
during  the  remainder  of  the  experiment. 


Practice  Exercise  5:  DON’T  BEGIN  UNTIL  I  TELL  YOU  TO  DO  SO!  Read  through  the 
following  exercise,  and  let  me  know  when  you  have  finished. 

In  this  exercise  you  will  pilot  the  MAV  around  the  main  roadway  that  forms  an  oval  in  the 
map.  Waypoints  will  be  visible  in  the  overhead  map  view  window.  Waypoints  are  used 
to  determine  the  correct  flight  path  of  the  MAV.  The  experimenter  will  explain  this  to  you 
in  more  detail  while  the  MAV  completes  the  mission  autonomously.  If  you  do  not  follow 
directions  or  if  you  crash,  you  will  be  required  to  restart  this  exercise  from  the  beginning. 
Be  careful  NOT  to  hit  the  poles! 

1)  The  experimenter  will  load  and  run  this  mission  autonomously  and  will  point  out 
the  Landing  Zone  (LZ)  on  the  (H)  building. 

2)  After  the  autonomous  mission  finishes,  the  simulation  will  be  reset. 

3)  You  must  now  manually  pilot  the  MAV  around  the  gray  pathway  while  remaining 
to  the  left  of  the  four  red  poles  and  then  land  in  the  correct  LZ. 

4)  When  ready,  press  OK  (button  10)  to  start  the  timer. 

5)  Execute  the  Take-off  command. 

6)  Complete  one  lap  around  the  four  red  poles  and  stay  over  the  gray  path. 

7)  Land  on  the  (H)  building. 

8)  This  exercise  must  be  completed  in  3  minutes  50  seconds  (3:50)  or  less. 
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Practice  Exercise  6:  DON’T  BEGIN  UNTIL  I  TELL  YOU  TO  DO  SO!  Read  through  the 
following  exercise,  and  let  me  know  when  you  have  finished. 

In  this  exercise  you  have  obstacles  to  navigate.  You  will  also  take  two  snapshots  at  the 
end  of  the  run.  Complete  the  mission  by  flying  through  the  series  of  red  and  green 
poles,  and  then  return  to  your  start  point  to  take  the  snapshots.  If  you  do  not  follow 
directions  or  if  you  crash,  you  will  be  required  to  restart  this  exercise  from  the  beginning. 
Be  careful  NOT  to  hit  the  poles! 


1)  The  experimenter  will  load  and  run  this  mission  autonomously.  Observe  how  the 
MAV  passes  to  the  right  of  all  green  poles  and  to  the  left  of  all  red  poles. 

2)  At  the  end  of  the  run  you  will  see  the  command-control  (C2)  vehicle  parked  on 
the  sidewalk.  (This  is  you!  You  control  the  MAV  from  this  position  inside  the 
vehicle.) 

3)  You  must  now  complete  the  course  manually  with  a  few  additional  instructions: 
After  you  finish  navigating  around  the  poles,  you  will  need  to  take 
snapshots  of  the  C2  vehicle  with  both  cameras  before  landing  the  MAV. 

4)  When  ready,  press  OK  (button  10)  and  then  execute  Take-off. 

5)  Complete  the  obstacle  course. 

6)  Take  snapshot  of  C2  vehicle  from  camera  1. 

7)  Take  snapshot  of  C2  vehicle  from  camera  2. 

8)  Land  -  but  do  NOT  land  on  the  C2  vehicle. 

9)  This  exercise  must  be  completed  in  5  minutes  (5:00)  or  less. 


You  will  now  complete  a  short  computer-based  questionnaire  and  a  paper-based 
training  questionnaire.  The  experimenter  will  explain  this  and  give  you 
instructions  at  this  time.  Then  you  will  begin  a  series  of  5  missions. 


Mission  Protocol 

There  will  be  5  missions  for  you  to  complete  during  this  portion  of  the  study.  The 
experimenter  will  instruct  you  on  mission  requirements  and  provide  any  documentation 
necessary.  If  you  are  unsure  of  any  of  these  requirements,  please  ask  for  clarification. 
Once  you  begin  a  mission,  the  experimenter  will  have  very  limited  interaction  with  you. 
He/she  will  not  be  able  to  answer  questions  on  mission  requirements  once  you  execute 
the  Take-off  command.  Please  ask  any  questions  beforehand. 
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Mission  1 


This  mission  is  a  repeat  of  practice  exercise  #5,  where  you  piloted  the  MAV  around  the 
gray  pathway  while  remaining  to  the  left  of  the  four  red  poles.  The  difference  in  this 
mission  is  that  there  will  be  no  waypoints  on  the  satellite  view  map.  You  need  to  try  to 
complete  this  mission  as  quickly  as  possible  but  also  without  any  collisions  -if  you  have 

a  collision  or  if  you  deviate  from  the  course,  you  will  be  required  to  redo  the  mission 

from  the  beginning  until  you  complete  it  without  a  collision. 


DON’T  BEGIN  UNTIL  I  TELL  YOU  TO  DO  SO!  Read  through  the  following  exercise, 
and  let  me  know  when  you  have  finished. 

1)  You  will  manually  pilot  the  MAV  around  the  gray  pathway  while  remaining  to  the 
left  of  the  four  red  poles,  and  then  land  on  the  (H)  building. 

2)  When  ready,  press  OK  to  start  the  timer. 

3)  Execute  the  Take-off  command. 

4)  Complete  one  lap  around  the  four  red  poles  and  stay  over  the  gray  path. 

5)  Land  on  the  (H)  building. 


Complete  the  computer-based  questionnaire  at  this  time,  and  then  proceed  to 
Mission  2. 
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Mission  2 


This  mission  repeats  practice  exercise  #6,  where  you  navigated  a  series  of  red  and 
green  poles.  You  will  also  take  two  snapshots  of  the  C2  vehicle  at  the  end  of  the  run. 
The  difference  in  this  mission  is  that  there  will  be  no  waypoints  on  the  satellite  view 
map.  Complete  the  mission  by  flying  through  the  series  of  red  and  green  poles,  and 
then  return  to  your  start  point  to  take  the  snapshots  of  the  C2  vehicle.  You  need  to  try  to 
complete  this  mission  as  quickly  as  possible  but  also  without  any  collisions  -if  you  have 

a  collision  or  if  you  deviate  from  the  course,  you  will  be  required  to  redo  the  mission 

from  the  beginning  until  you  complete  it  without  a  collision. 


DON’T  BEGIN  UNTIL  I  TELL  YOU  TO  DO  SO!  Read  through  the  following  exercise, 
and  let  me  know  when  you  have  finished. 

1)  You  must  complete  the  obstacle  course  manually. 

2)  After  you  finish  navigating  around  the  poles  (right  of  the  green  poles  and  left  of 
the  red  poles),  you  will  need  to  take  snapshots  of  the  C2  vehicle  with  both 
cameras. 

3)  When  ready,  press  OK  and  then  execute  Take-off. 

4)  Complete  the  obstacle  course. 

5)  Take  snapshot  of  the  C2  vehicle  with  camera  1 . 

6)  Take  snapshot  of  the  C2  vehicle  with  camera  2. 

7)  Land,  but  do  NOT  land  on  the  C2  vehicle. 


Complete  the  computer-based  questionnaire  at  this  time,  and  then  proceed  to 
Mission  3. 
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Mission  3 


DON’T  BEGIN  UNTIL  I  TELL  YOU  TO  DO  SO!  Read  through  the  following  exercise, 
and  let  me  know  when  you  have  finished. 

This  mission  involves  using  the  MAV  to  do  reconnaissance  work.  You  will  get  a  handout 
titled  "Mission  3  Intel  &  Recon.”  Review  this  with  the  experimenter,  and  then  complete 
the  required  tasks.  This  is  primarily  a  target  identification  mission.  The  experimenter 
may  ask  you  for  situational  updates  during  this  mission. 

1)  Review  the  "Mission  3  Intel  &  Recon”  handout. 

2)  The  experimenter  will  load  the  mission  files  and  scenario. 

3)  You  will  have  a  limited  time  to  identify  as  many  targets  as  possible  on  the  map. 

4)  Positive  ID  can  only  be  achieved  by  taking  snapshots  of  each  entity  with  both  the 
forward  and  downward  cameras,  and  each  entity  must  be  centered  in  the  frame 
so  that  the  center  ( )  overlay  is  touching  part  of  the  entity. 

5)  When  ready  press  OK  to  begin  the  mission  and  start  the  timer. 

6)  Immediately  begin  looking  for  entities  to  identify  via  the  cameras. 

7)  The  experimenter  will  tell  you  when  time  has  expired. 


Complete  the  computer-based  questionnaire  at  this  time  and  then  proceed  to 
Mission  4. 
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Mission  4 


This  mission  involves  using  the  MAV  to  navigate  around  obstacles.  Complete  the 
mission  by  piloting  the  MAV  around  the  roadway,  flying  through  the  series  of  red  and 
green  poles,  and  then  landing  in  the  designated  area.  You  must  fly  to  the  left  of  the 
red  poles  and  to  the  right  of  the  green  poles.  Also,  be  sure  to  stay  close  to  the 
pathway!  You  need  to  try  to  complete  this  mission  as  quickly  as  possible  but  also 
without  any  collisions  or  deviations  from  the  course  -if  you  have  a  collision,  you  will  be 

required  to  redo  the  mission  from  the  beginning  until  you  complete  it  without  a  collision. 

DON’T  BEGIN  UNTIL  I  TELL  YOU  TO  DO  SO!  Read  through  the  following  exercise, 
and  let  me  know  when  you  have  finished. 


1)  The  experimenter  will  load  and  run  this  mission  autonomously  with  waypoints 
visible.  Observe  how  the  MAV  passes  to  the  right  of  all  green  poles  and  to  the 
left  of  all  red  poles. 

2)  You  must  now  complete  the  course  manually.  You  must  navigate  the  obstacle 
course  without  crashing  before  you  can  move  on  to  the  final  mission. 

3)  When  ready,  press  OK  and  then  execute  Take-off. 

4)  Complete  the  obstacle  course. 

5)  Land  in  the  designated  area. 

6)  This  exercise  must  be  completed  without  crashing  in  as  many  tries  as  necessary. 
There  is  no  time  limit. 


Complete  the  computer-based  questionnaire  at  this  time  and  then  proceed  to 
Mission  5. 
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Mission  5 


DON’T  BEGIN  UNTIL  I  TELL  YOU  TO  DO  SO!  Read  through  the  following  exercise, 
and  let  me  know  when  you  have  finished. 

This  mission  involves  using  the  MAV  to  do  more  reconnaissance  work.  You  will  get  a 
handout  titled  “Mission  5  Intel  &  Recon.”  Review  this  with  the  experimenter,  and  then 
complete  the  required  tasks.  This  is  primarily  a  target  identification  mission.  The 
experimenter  may  ask  you  for  situational  updates  during  this  mission. 

1)  Review  the  “Mission  5  Intel  &  Recon”  handout. 

2)  The  experimenter  will  load  the  mission  files  and  scenario. 

3)  You  will  have  a  limited  time  to  identify  as  many  targets  as  possible  on  the  map. 

4)  Positive  ID  can  only  be  achieved  by  taking  snapshots  of  each  entity  with  the 
forward  camera  so  that  the  person’s  face  is  visible  and  identifiable. 

5)  When  ready  press  OK  to  begin  the  mission  and  start  the  timer. 

6)  Immediately  begin  looking  for  entities  to  identify  via  the  cameras. 

7)  The  experimenter  will  tell  you  when  time  has  expired. 


Complete  the  computer-based  questionnaire  at  this  time  along  with  a  post-test, 
and  then  you  may  proceed  to  your  final  debriefing  session. 
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Appendix  D 


Demographics  Questionnaire 


1 .  Participant  Number 


2.  Today’s  Date 


3.  Year  of  Birth 


4.  Gender 

o 

Male 

0 

Female 

5.  Have  you  graduated  from  High  School? 

o 

Yes  o 

No 

6.  Which  hand  do  you  write  with? 

o 

Right 

o 

Left 

7.  Is  your  vision  in  each  eye  correctable  to  20/20? 

o 

Yes  o 

No 

8.  To  your  knowledge  are  you  color  blind? 

o 

Yes 

o 

No 

9.  Do  you  own  or  have  access  to  a  computer? 

o 

Yes 

No 

10.  If  yes,  how  often  do  you  use  a  computer? 

Daily 

o 

Several  times  a  week 

o 

Occasionally 

o 

Never 

o 

1 1 .  Estimate  how  many  hours  per  week  you  use  a  computer. 

Never 

o 

1  -5  hours 

o 

5-10  hours 

o 

1 0-20  hours 

o 

20-30  hours 

o 

30-40  hours 

o 

40  +  hours 

o 

12.  How  would  you  rate  your  computer  skills? 

Novice/Beginner 

o 

Intermediate 

o 

Expert 

o 

13.  Do  you  use  the  Internet? 

o 

Yes 

No 
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14.  Do  you  have  any  previous  remote  control  (R/C) 
experience? 

o 

Yes 

O 

No 

15.  Do  you  own  or  use  a  video  game  system? 

o 

Yes 

O 

No 

1 6.  How  would  you  rate  your  video  game  skills? 

Novice/Beginner 

o 

Intermediate 

o 

Expert 

0 

1 7.  How  often  do  you  play  video  games? 

Daily 

o 

Several  times  a  week 

0 

Occasionally 

0 

Never 

0 

1 1 .  Estimate  how  many  hours  per  week  you  play  video  games. 

Never 

o 

1  -5  hours 

o 

5-10  hours 

0 

1 0-20  hours 

0 

20-30  hours 

0 

30-40  hours 

o 

40  +  hours 

0 

13.  Select  any  of  the  following  game  consoles  that  you  either  own  or  have  used  on  a  regular 
basis. 

□  Microsoft  XBOX 

□  Sony  Playstation 

□  Sony  Playstation  2 

□  XBOX  360 

□  Sega  Dreamcast 

□  Nintendo  Gamecube 

□  Super  Nintendo 

□  PC  game  system 

14.  How  would  you  rate  your  video  game  skills? 

Novice/Beginner 

o 

Intermediate 

o 

Expert 

o 

15.  How  many  days  in  the  past  week  have  you  played  video  games? 

0 

o 

1 

o 

2 

o 

3 

o 

4 

o 

5 

o 

6 

o 

7 

o 
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16.  Estimate  how  many  hours  per  day  you  play  video  games  on  average. 

0 

o 

1 

o 

2 

o 

3 

o 

4 

o 

5 

o 

6 

o 

7 

o 

8 

o 

More  than  8  hours  per  day 

o 

D-3 


Appendix  E 


MAV  Training  Evaluation  Worksheet 
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To  ensure  that  you  have  a  basic  grasp  of  the  MAV  pilot  interface  and  the  available  flight 
commands,  please  complete  the  rol lowing  exercise. 

Each  of  the  critical  features  of  the  user  interface  is  labeled  above  with  letters  A  -  N. 
Every  letter  must  be  used  only  once,  so  choose  the  best  answer.  Enter  the 
corresponding  letter  in  the  blank  following  each  of  the  item  descriptions  below: 


Altimeter _ 

Mission  Timer _ 

MAV  Rotational  Icons  _ 

Heading  Tape  _ 

Satellite  Map  View _ 

Current  MAV  Altitude  _ 
Forward  Camera  Image 


Camera  Selection  Icons 

Altitude  Control _ 

Take-off  and  Land  Icon 

Halt  Movement  Icon _ 

MAV  Location  on  Map  _ 
Ground  Level  Indicator  . 
Air  Speed  Indicator _ 
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MAV  USABILITY  QUESTIONNAIRE  (Game  controller  or  mouse) 

Circle  the  number  that  best  describes  your  reaction  between  the  2  extremes  given. 


1 )  The  system  I  worked  with  was 
wonderful 

12  3  4 

5 

6 

7 

8 

9 

terrible 

10 

2)  The  system  I  worked  with  was 
easy 

12  3  4 

5 

6 

7 

8 

9 

difficult 

10 

3)  I  found  this  experience 
satisfying 

12  3  4 

5 

6 

7 

8 

9 

frustrating 

10 

4)  The  system  1  worked  with  seemed 

Capable  of  doing  the 

exercises 

1  2  3  4  5  6  7 


Unable  to  do  the 
exercises 

8  9  10 


5)  I  found  this  experience 

stimulating  dull 

123456789  10 


6)  The  system  I  worked  with  was 

flexible  rigid 

123456789  10 


7)  The  functions  of  the  on-screen  manual  control  buttons  (on  the  control  pad)  were 
clear  confusing 


123456789  10 


8)  The  functionality  of  the  buttons  for  switching  between  camera  views  was 
clear 

123456789 


confusing 

10 
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8 


confusing 
9  10 


9)  Organization  of  information  on  the  video  display  screen  was 
clear 

1  2  3  4  5  6  7 


10)  Current  status  (such  as  Taking  off,  Landing,  Grounded)  messages  were 
adequate 


1  2  3  4  5  6  7  8 


9 


inadequate 

10 


1 1)  When  controlling  the  MAV  in  flight  using  the  joystick,  the  device  was 
easy  to  use  difficult  to  use 

1  23456789  10 


12)  As  1  progressed  through  the  missions  using  the  joystick,  my  hands  and/or  wrists  became 


fatigued 

Never 

Always 

1  2 

3 

4 

5 

6 

7 

8 

9  10 

13)  When  using  the  joystick  to  enter  flight  commands  to  the  MAV,  maintaining  awareness  of 
individual  mission  objectives  was 

easy  difficult 

1  23456789  10 


14)  When  using  the  LEFT  thumb  stick  (forward,  back,  left  &  right  movements)  to  move  the  air 
vehicle  while  in  manual  control,  the  air  vehicle  reacted  as  I  expected, 
always  never 

123456789  10 


1 5)  Using  the  RIGHT  thumb  stick  (up,  down  &  rotation)  to  move  the  air  vehicle  while  in  manual 
control,  the  air  vehicle  reacted  as  I  expected. 

always  never 

123456789  10 


16)  When  using  the  joystick  to  stop  the  motion  of  the  air  vehicle  while  in  manual  control,  the  air 
vehicle  reacted  as  I  expected. 

always  never 

123456789  10 
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17)  When  using  the  joystick  to  switch  between  camera  views  (highlighting  the  camera  icon 
located  on  the  task  bar),  the  display  reacted  as  I  expected. 

always  never 

123456789  10 


For  Joystick  condition  skip  to  question  18  on  next  page. 


FOR  MOUSE  CONDITION  ONLY  Substitute  the  following  for  Questions  11  -17: 


11)  When  controlling  the  MA  V  in  flight  using  the  mouse,  the  device  was 
easy  to  use 

1  2  3  4  5  6  7  8 


difficult  to  use 
9  10 


12)  As  I  progressed  through  the  missions  using  the  mouse,  my  hand  and/or  wrist  became 
fatigued 

Never  Always 

123456789  10 


13)  When  using  the  mouse  to  enter  flight  commands  to  the  MA  V,  maintaining  awareness  of 
individual  mission  objectives  was 

Easy  Difficult 

123456789  10 


14)  Using  the  LEFT/RIGHT  buttons  to  move  MA  V  while  in  manual  control,  the  air  vehicle 
reacted  as  I  expected. 

Always  Never 

123456789  10 


15)  Using  the  ROTATE  buttons  to  move  the  MA  V  while  in  manual  control,  the  air  vehicle 
reacted  as  I  expected. 

Always  Never 

123456789  10 


16)  Using  the  Stop  (X)  button  to  stop  the  motion  of  the  MA  V  while  in  manual  control,  the  air 
vehicle  reacted  as  I  expected. 

Always  Never 

123456789  10 


1 7)  When  using  the  icon  to  switch  between  camera  views  (located  on  the  task  bar),  the  display 
reacted  as  I  expected. 

Always  Never 


1  23456789  10 


1 8)  While  using  the  camera  to  take  snapshots  of  targets,  centering  the  target  so  it  was  aligned 
with  the  ( )  overlay  was 

easy  difficult 

1  23456789  10 


1 9)  When  piloting  the  MAY  in  manual  control,  determining  the  current  heading  of  the  MAY 


from  the  360-degree  directional  heading  tape  was 
clear 

1  2  3  4  5  6 

7 

8 

confusing 
9  10 

20)  It  was  clear  when  the  air  vehicle  had  landed 
always 

1  2  3  4  5  6 

7 

8 

never 

9  10 

21)  The  system  provided  adequate  feedback  when  I  issued  a  command  to  the  MAY. 


always 

1  2  3  4  5  6 

7 

8 

9 

never 

10 

22)  The  OCU  interface  keeps  you  informed  about  what  it  is  happening 
always 

1  2  3  4  5  6  7  8 

9 

never 

10 

23)  Learning  to  operate  the  system  was 
easy 

1  2  3  4  5  6 

7 

8 

9 

difficult 

10 

24)  Remembering  names  and  use  of  commands  was 
easy 

1  2  3  4  5  6 

7 

8 

9 

difficult 

10 
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25)  Tasks  could  be  performed  in  a  straightforward  manner 

always  never 

1  23456789  10 


26)  Training  materials  were 

clear  confusing 

1  23456789  10 


27)  After  this  training,  I  am  ready  use  this  system  to  fly  a  real  air  vehicle 

very  confident  not  at  all 

confident 

123456789  10 

28)  The  flight  speed  of  the  MAV  was 

fast  enough  too  slow 

123456789  10 


29)  The  responsiveness  of  the  system  to  my  input  was 

Fast  enough  Too  slow 

123456789  10 


30)  System  reliability  was 
reliable 

1  2  3  4  5  6  7  8 


unreliable 
9  10 


31)  Correcting  your  mistakes  was 
easy 

1  2  3  4  5  6  7  8 


difficult 
9  10 


32)  Both  experienced  and  inexperienced  users’  needs  were  taken  into  consideration 
always  never 

123456789  10 
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